Oreopoulos, Philip; Petronijevic, Uros (2019). National Bureau of Economic Research. Retrieved from: https://eric.ed.gov/?id=ED604421
Reviewed: March 2022
This review may not reflect the full body of research evidence for this intervention.
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|
|---|---|---|---|---|---|---|---|---|---|
|
Final Grade Average (percent) Year 1 |
Nudging intervention vs. Business as usual |
0 Days |
online and one-way text coaching vs. comparison (personality test);
|
67.70 |
67.60 |
No |
-- | ||
|
Final Grade Average (percent) Year 1 |
Nudging intervention vs. Business as usual |
0 Days |
online and face-to-face coaching vs. comparison (personality test);
|
67.69 |
67.60 |
No |
-- | ||
|
Final Grade Average (percent) Year 1 |
Nudging intervention vs. Business as usual |
0 Days |
online and two-way text coaching vs. comparison (personality test);
|
67.40 |
67.60 |
No |
-- | ||
| Show Supplemental Findings | |||||||||
|
Final Grade Average (percent) Year 3 |
Nudging intervention vs. Business as usual |
2 Years |
online and face-to-face coaching vs. comparison (personality test);
|
75.30 |
71.00 |
No |
-- | ||
|
Final Grade Average (percent) Year 2 |
Nudging intervention vs. Business as usual |
1 Year |
online and face-to-face coaching vs. comparison (personality test);
|
70.38 |
69.20 |
No |
-- | ||
|
Final Grade Average (percent) Year 3 |
Nudging intervention vs. Business as usual |
2 Years |
online and one-way text coaching vs. comparison (personality test);
|
71.42 |
71.00 |
No |
-- | ||
|
Final Grade Average (percent) Year 2 |
Nudging intervention vs. Business as usual |
1 Year |
online and one-way text coaching vs. comparison (personality test);
|
69.20 |
69.20 |
No |
-- | ||
|
Final Grade Average (percent) Year 2 |
Nudging intervention vs. Business as usual |
1 Year |
online and two-way text coaching vs. comparison (personality test);
|
69.10 |
69.20 |
No |
-- | ||
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|
|---|---|---|---|---|---|---|---|---|---|
|
persisted year 2 |
Nudging intervention vs. Business as usual |
3 Months |
online and face-to-face coaching vs. comparison (personality test);
|
84.70 |
80.40 |
No |
-- | ||
|
year 1 credits earned |
Nudging intervention vs. Business as usual |
0 Days |
online and one-way text coaching vs. comparison (personality test);
|
3.10 |
3.10 |
No |
-- | ||
|
persisted year 2 |
Nudging intervention vs. Business as usual |
3 Months |
online and two-way text coaching vs. comparison (personality test);
|
80.30 |
80.40 |
No |
-- | ||
|
year 1 credits earned |
Nudging intervention vs. Business as usual |
0 Days |
online and two-way text coaching vs. comparison (personality test);
|
3.06 |
3.10 |
No |
-- | ||
|
year 1 credits earned |
Nudging intervention vs. Business as usual |
0 Days |
online and face-to-face coaching vs. comparison (personality test);
|
3.06 |
3.10 |
No |
-- | ||
|
persisted year 2 |
Nudging intervention vs. Business as usual |
3 Months |
online and one-way text coaching vs. comparison (personality test);
|
79.20 |
80.40 |
No |
-- | ||
| Show Supplemental Findings | |||||||||
|
year 3 credits earned |
Nudging intervention vs. Business as usual |
2 Years |
online and face-to-face coaching vs. comparison (personality test);
|
4.32 |
2.90 |
Yes |
|
||
|
persisted year 3 |
Nudging intervention vs. Business as usual |
15 Months |
online and face-to-face coaching vs. comparison (personality test);
|
78.00 |
72.70 |
No |
-- | ||
|
year 2 credits earned |
Nudging intervention vs. Business as usual |
1 Year |
online and face-to-face coaching vs. comparison (personality test);
|
3.29 |
3.00 |
No |
-- | ||
|
year 3 credits earned |
Nudging intervention vs. Business as usual |
2 Years |
online and one-way text coaching vs. comparison (personality test);
|
3.03 |
2.90 |
Yes |
|
||
|
year 2 credits earned |
Nudging intervention vs. Business as usual |
1 Year |
online and one-way text coaching vs. comparison (personality test);
|
3.05 |
3.00 |
No |
-- | ||
|
persisted year 3 |
Nudging intervention vs. Business as usual |
15 Months |
online and two-way text coaching vs. comparison (personality test);
|
73.70 |
72.70 |
No |
-- | ||
|
year 2 credits earned |
Nudging intervention vs. Business as usual |
1 Year |
online and two-way text coaching vs. comparison (personality test);
|
2.96 |
3.00 |
No |
-- | ||
|
persisted year 3 |
Nudging intervention vs. Business as usual |
15 Months |
online and one-way text coaching vs. comparison (personality test);
|
71.50 |
72.70 |
No |
-- | ||
Characteristics of study sample as reported by study author.
The study was conducted in introductory economics courses offered at three University of Toronto campuses: UofT at St. George (UTSG, downtown), UofT at Mississauga (UTM, suburban, commuter campus), and UofT at Scarborough (UTSC, semi-urban neighborhood of Toronto, commuter campus) between 2014 and 2018.
The sample included five cohorts (2014-2018) of students taking a first-year economics course at any of the three UofT campuses. Within campus and year, students were randomized to one of several interventions or to the comparison condition. This review focuses on three of the six interventions studied: (1) online and one-way text coaching; (2) online and two-way text coaching; and (3) online and face-to-face coaching. The randomized sample size for the entire study was 24,772 students. A total of 10,809 students are included in the analyses reported in this review. The randomized samples corresponding to the contrasts reported in this review are as follows: (1) online and one-way text coaching, 2,709 students; (2) online and two-way text coaching, 2,575 students; and (3) online and face-to-face coaching, 90 students. The comparison sample across all contrasts in years in which one of the three interventions was implemented and yielded impact estimates (2014-2017) was 5,435 students. The authors did not report data on race or ethnicity, but 31.4% of the sample in this review were first-generation college students, 52% were female, and students were 20.3 years old on average.
This review examines the effects of three interventions. Three other interventions (goal setting, mindset, and online coaching only) are included in the study but are not part of this review. Online coaching plus one-way text coaching: Students in this group participated in an online module (about 60 minutes) that included goal-setting, mindset, and college study skills such as strategies for effective studying, seeking help, motivation, patience, and reaching out for help. Over the course of the study, two versions of online coaching were implemented. Both had the objective of helping students be more successful in college. In the first version implemented in 2015, students completed a writing exercise requiring them to think about their futures in terms of professional and personal aspirations as well as their academic expectations for the current year. In the second version implemented in 2016-17, students set goals and engaged in ‘mindset’ activities that attempted to help students develop a more positive perspective on their academic endeavors. Students in this group then had the opportunity to receive follow-up coaching via text message or email throughout the rest of the academic year. The messages this group received were designed to not elicit a response. Themes of the messages were academic and study preparation, university resource information, and motivation/encouragement. Online coaching plus two-way text coaching: This intervention was similar to the one-way text coaching intervention (online module followed by text messaging) with the main difference being that the text messages were more personalized (they were sent by upper-year undergraduate coaches) and encouraged responses, questions, and progress check-ins. Coaches were expected to communicate at least weekly with their students as well as follow up on issues specific to individuals’ needs. Online and face-to-face coaching: Students in this group completed the online module and were offered personal coaches who met with them weekly for 30 minutes to an hour as well as via Skype, email, or text between meetings. The coaches offered guidance across a variety of topics including major selection, campus employment opportunities, specific coursework questions, and students’ anxieties and general emotional well-being. Not all interventions were implemented on all campuses in all years. In 2015-16, one-way text coaching and face-to-face coaching were implemented on all campuses. In 2016-17, one-way text coaching and face-to-face coaching were implemented at the UTM campus and two-way text coaching was implemented on the UTSG campus. In 2017-18, two-way text coaching was implemented on the UTSG and UTM campuses.
The comparison group received a set of questions that required about 45-60 minutes to complete. The questions were personality tests and questions about risk tolerance and time preferences. The authors then emailed students a brief report describing the results of their personality test and how to interpret them.
An indicator of the effect of the intervention, the improvement index can be interpreted as the expected change in percentile rank for an average comparison group student if that student had received the intervention.
For more, please see the WWC Glossary entry for improvement index.
An outcome is the knowledge, skills, and attitudes that are attained as a result of an activity. An outcome measures is an instrument, device, or method that provides data on the outcome.
A finding that is included in the effectiveness rating. Excluded findings may include subgroups and subscales.
The sample on which the analysis was conducted.
The group to which the intervention group is compared, which may include a different intervention, business as usual, or no services.
The timing of the post-intervention outcome measure.
The number of students included in the analysis.
The mean score of students in the intervention group.
The mean score of students in the comparison group.
The WWC considers a finding to be statistically significant if the likelihood that the finding is due to chance alone, rather than a real difference, is less than five percent.
The WWC reviews studies for WWC products, Department of Education grant competitions, and IES performance measures.
The name and version of the document used to guide the review of the study.
The version of the WWC design standards used to guide the review of the study.
The result of the WWC assessment of the study. The rating is based on the strength of evidence of the effectiveness of the intervention. Studies are given a rating of Meets WWC Design Standards without Reservations, Meets WWC Design Standards with Reservations, or >Does Not Meet WWC Design Standards.
A related publication that was reviewed alongside the main study of interest.
Study findings for this report.
Based on the direction, magnitude, and statistical significance of the findings within a domain, the WWC characterizes the findings from a study as one of the following: statistically significant positive effects, substantively important positive effects, indeterminate effects, substantively important negative effects, and statistically significant negative effects. For more, please see the WWC Handbook.
The WWC may review studies for multiple purposes, including different reports and re-reviews using updated standards. Each WWC review of this study is listed in the dropdown. Details on any review may be accessed by making a selection from the drop down list.
Tier 1 Strong indicates strong evidence of effectiveness,
Tier 2 Moderate indicates moderate evidence of effectiveness, and
Tier 3 Promising indicates promising evidence of effectiveness,
as defined in the
non-regulatory guidance for ESSA
and the regulations for ED discretionary grants (EDGAR Part 77).
Resources
Newsflash
Contact Us
Facebook
Twitter
YouTube