WWC review of this study

The remarkable unresponsiveness of college students to nudging and what we can learn from it. NBER Working Paper No. 26059.

Oreopoulos, P., & Petronijevic, U. (2019). National Bureau of Economic Research. Retrieved from: https://eric.ed.gov/?id=ED604421

  • Randomized Controlled Trial
     examining 
    10,809
     Students
    , grade
    PS

Reviewed: March 2022

At least one statistically significant positive finding
Meets WWC standards without reservations
Postsecondary Academic Achievement outcomes—Indeterminate effects found
Outcome
measure
Comparison Period Sample Intervention
mean
Comparison
mean
Significant? Improvement
    index
Evidence
tier

Final Grade Average (percent) Year 1

Nudging intervention vs. Business as usual

0 Days

online and one-way text coaching vs. comparison (personality test);
4,904 students

67.70

67.60

No

--
More Outcomes

Final Grade Average (percent) Year 1

Nudging intervention vs. Business as usual

0 Days

online and face-to-face coaching vs. comparison (personality test);
2,448 students

67.69

67.60

No

--

Final Grade Average (percent) Year 1

Nudging intervention vs. Business as usual

0 Days

online and two-way text coaching vs. comparison (personality test);
2,020 students

67.40

67.60

No

--
Show Supplemental Findings

Final Grade Average (percent) Year 3

Nudging intervention vs. Business as usual

2 Years

online and face-to-face coaching vs. comparison (personality test);
1,409 students

75.30

71.00

No

--

Final Grade Average (percent) Year 2

Nudging intervention vs. Business as usual

1 Year

online and face-to-face coaching vs. comparison (personality test);
2,127 students

70.38

69.20

No

--

Final Grade Average (percent) Year 3

Nudging intervention vs. Business as usual

2 Years

online and one-way text coaching vs. comparison (personality test);
2,867 students

71.42

71.00

No

--

Final Grade Average (percent) Year 2

Nudging intervention vs. Business as usual

1 Year

online and one-way text coaching vs. comparison (personality test);
4,244 students

69.20

69.20

No

--

Final Grade Average (percent) Year 2

Nudging intervention vs. Business as usual

1 Year

online and two-way text coaching vs. comparison (personality test);
1,806 students

69.10

69.20

No

--
Progressing in college outcomes—Indeterminate effects found
Outcome
measure
Comparison Period Sample Intervention
mean
Comparison
mean
Significant? Improvement
    index
Evidence
tier

persisted year 2

Nudging intervention vs. Business as usual

3 Months

online and face-to-face coaching vs. comparison (personality test);
2,564 students

84.70

80.40

No

--
More Outcomes

year 1 credits earned

Nudging intervention vs. Business as usual

0 Days

online and one-way text coaching vs. comparison (personality test);
5,183 students

3.10

3.10

No

--

persisted year 2

Nudging intervention vs. Business as usual

3 Months

online and two-way text coaching vs. comparison (personality test);
5,536 students

80.30

80.40

No

--

year 1 credits earned

Nudging intervention vs. Business as usual

0 Days

online and two-way text coaching vs. comparison (personality test);
5,536 students

3.06

3.10

No

--

year 1 credits earned

Nudging intervention vs. Business as usual

0 Days

online and face-to-face coaching vs. comparison (personality test);
2,564 students

3.06

3.10

No

--

persisted year 2

Nudging intervention vs. Business as usual

3 Months

online and one-way text coaching vs. comparison (personality test);
5,183 students

79.20

80.40

No

--
Show Supplemental Findings

year 3 credits earned

Nudging intervention vs. Business as usual

2 Years

online and face-to-face coaching vs. comparison (personality test);
1,817 students

4.32

2.90

Yes

 
 
27

persisted year 3

Nudging intervention vs. Business as usual

15 Months

online and face-to-face coaching vs. comparison (personality test);
2,564 students

78.00

72.70

No

--

year 2 credits earned

Nudging intervention vs. Business as usual

1 Year

online and face-to-face coaching vs. comparison (personality test);
2,564 students

3.29

3.00

No

--

year 3 credits earned

Nudging intervention vs. Business as usual

2 Years

online and one-way text coaching vs. comparison (personality test);
3,792 students

3.03

2.90

Yes

 
 
3

year 2 credits earned

Nudging intervention vs. Business as usual

1 Year

online and one-way text coaching vs. comparison (personality test);
5,183 students

3.05

3.00

No

--

persisted year 3

Nudging intervention vs. Business as usual

15 Months

online and two-way text coaching vs. comparison (personality test);
2,230 students

73.70

72.70

No

--

year 2 credits earned

Nudging intervention vs. Business as usual

1 Year

online and two-way text coaching vs. comparison (personality test);
2,230 students

2.96

3.00

No

--

persisted year 3

Nudging intervention vs. Business as usual

15 Months

online and one-way text coaching vs. comparison (personality test);
5,183 students

71.50

72.70

No

--


Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.

Characteristics of study sample as reported by study author.


  • Female: 52%
    Male: 48%

  • Suburban, Urban
    • B
    • A
    • C
    • D
    • E
    • F
    • G
    • I
    • H
    • J
    • K
    • L
    • P
    • M
    • N
    • O
    • Q
    • R
    • S
    • V
    • U
    • T
    • W
    • X
    • Z
    • Y
    • a
    • h
    • i
    • b
    • d
    • e
    • f
    • c
    • g
    • j
    • k
    • l
    • m
    • n
    • o
    • p
    • q
    • r
    • s
    • t
    • u
    • x
    • w
    • y

    International

Setting

The study was conducted in introductory economics courses offered at three University of Toronto campuses: UofT at St. George (UTSG, downtown), UofT at Mississauga (UTM, suburban, commuter campus), and UofT at Scarborough (UTSC, semi-urban neighborhood of Toronto, commuter campus) between 2014 and 2018.

Study sample

The sample included five cohorts (2014-2018) of students taking a first-year economics course at any of the three UofT campuses. Within campus and year, students were randomized to one of several interventions or to the comparison condition. This review focuses on three of the six interventions studied: (1) online and one-way text coaching; (2) online and two-way text coaching; and (3) online and face-to-face coaching. The randomized sample size for the entire study was 24,772 students. A total of 10,809 students are included in the analyses reported in this review. The randomized samples corresponding to the contrasts reported in this review are as follows: (1) online and one-way text coaching, 2,709 students; (2) online and two-way text coaching, 2,575 students; and (3) online and face-to-face coaching, 90 students. The comparison sample across all contrasts in years in which one of the three interventions was implemented and yielded impact estimates (2014-2017) was 5,435 students. The authors did not report data on race or ethnicity, but 31.4% of the sample in this review were first-generation college students, 52% were female, and students were 20.3 years old on average.

Intervention Group

This review examines the effects of three interventions. Three other interventions (goal setting, mindset, and online coaching only) are included in the study but are not part of this review. Online coaching plus one-way text coaching: Students in this group participated in an online module (about 60 minutes) that included goal-setting, mindset, and college study skills such as strategies for effective studying, seeking help, motivation, patience, and reaching out for help. Over the course of the study, two versions of online coaching were implemented. Both had the objective of helping students be more successful in college. In the first version implemented in 2015, students completed a writing exercise requiring them to think about their futures in terms of professional and personal aspirations as well as their academic expectations for the current year. In the second version implemented in 2016-17, students set goals and engaged in ‘mindset’ activities that attempted to help students develop a more positive perspective on their academic endeavors. Students in this group then had the opportunity to receive follow-up coaching via text message or email throughout the rest of the academic year. The messages this group received were designed to not elicit a response. Themes of the messages were academic and study preparation, university resource information, and motivation/encouragement. Online coaching plus two-way text coaching: This intervention was similar to the one-way text coaching intervention (online module followed by text messaging) with the main difference being that the text messages were more personalized (they were sent by upper-year undergraduate coaches) and encouraged responses, questions, and progress check-ins. Coaches were expected to communicate at least weekly with their students as well as follow up on issues specific to individuals’ needs. Online and face-to-face coaching: Students in this group completed the online module and were offered personal coaches who met with them weekly for 30 minutes to an hour as well as via Skype, email, or text between meetings. The coaches offered guidance across a variety of topics including major selection, campus employment opportunities, specific coursework questions, and students’ anxieties and general emotional well-being. Not all interventions were implemented on all campuses in all years. In 2015-16, one-way text coaching and face-to-face coaching were implemented on all campuses. In 2016-17, one-way text coaching and face-to-face coaching were implemented at the UTM campus and two-way text coaching was implemented on the UTSG campus. In 2017-18, two-way text coaching was implemented on the UTSG and UTM campuses.

Comparison Group

The comparison group received a set of questions that required about 45-60 minutes to complete. The questions were personality tests and questions about risk tolerance and time preferences. The authors then emailed students a brief report describing the results of their personality test and how to interpret them.

 

Your export should download shortly as a zip archive.

This download will include data files for study and findings review data and a data dictionary.

Connect With the WWC

loading
back to top