WWC review of this study

Student coaching: How far can technology go?

Oreopoulos, P., & Petronijevic, U. (2018). Journal of Human Resources, 53(2), 299-329.

  • Randomized Controlled Trial
     examining 
    4,387
     Students
    , grade
    PS

Reviewed: May 2021

At least one finding shows strong evidence of effectiveness
At least one statistically significant positive finding
Meets WWC standards without reservations
Academic achievement outcomes—Statistically significant positive effects found
Outcome
measure
Comparison Period Sample Intervention
mean
Comparison
mean
Significant? Improvement
    index
Evidence
tier

GPA for Online + Coaching

Three interventions: online exercise, text message champaign, personal coaching service vs. Other intervention

0 Weeks

Full sample;
1,335 students

2.74

2.38

No

--
More Outcomes

All course grades for Online + Coaching

Three interventions: online exercise, text message champaign, personal coaching service vs. Other intervention

0 Weeks

Full sample;
1,335 students

70.11

67.27

Yes

 
 
8
 

GPA for Online + Text Messaging

Three interventions: online exercise, text message champaign, personal coaching service vs. Other intervention

0 Weeks

Full sample;
4,646 students

2.36

2.40

No

--

All course grades for Online + Text Messaging

Three interventions: online exercise, text message champaign, personal coaching service vs. Other intervention

0 Weeks

Full sample;
4,646 students

68.79

69.37

No

--
Show Supplemental Findings

Winter course grades for Online + Coaching

Three interventions: online exercise, text message champaign, personal coaching service vs. Other intervention

0 Weeks

Full sample;
1,335 students

75.35

68.39

No

--

Winter course grades for Online + Text Messaging

Three interventions: online exercise, text message champaign, personal coaching service vs. Other intervention

0 Weeks

Full sample;
4,646 students

68.66

69.42

No

--
Progressing in college outcomes—Statistically significant positive effects found
Outcome
measure
Comparison Period Sample Intervention
mean
Comparison
mean
Significant? Improvement
    index
Evidence
tier

College credits earned for online vs coaching

Three interventions: online exercise, text message champaign, personal coaching service vs. Other intervention

0 Weeks

Full sample;
1,335 students

4.19

3.65

No

--
More Outcomes

College credits failed for online vs coaching

Three interventions: online exercise, text message champaign, personal coaching service vs. Other intervention

0 Weeks

Full sample;
1,335 students

0.20

0.33

No

--

College credits failed for online vs text messaging

Three interventions: online exercise, text message champaign, personal coaching service vs. Other intervention

0 Weeks

Full sample;
4,646 students

0.35

0.33

No

--

College credits earned for online vs text messaging

Three interventions: online exercise, text message champaign, personal coaching service vs. Other intervention

0 Weeks

Full sample;
4,646 students

3.64

3.65

No

--


Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.

Characteristics of study sample as reported by study author.


  • 55% English language learners

  • Female: 53%
    Male: 47%

  • Urban
    • B
    • A
    • C
    • D
    • E
    • F
    • G
    • I
    • H
    • J
    • K
    • L
    • P
    • M
    • N
    • O
    • Q
    • R
    • S
    • V
    • U
    • T
    • W
    • X
    • Z
    • Y
    • a
    • h
    • i
    • b
    • d
    • e
    • f
    • c
    • g
    • j
    • k
    • l
    • m
    • n
    • o
    • p
    • q
    • r
    • s
    • t
    • u
    • x
    • w
    • y

    International
  • Race
    Other or unknown
    100%

Setting

The study was set in three campuses of the University of Toronto (U of T, Ontario, Canada): Mississauga and Scarborough, which are commuter campuses, and St. George, which is the main campus and is located in downtown Toronto and has a larger resident student population. The personal coaching intervention only took place on one of the three campuses.

Study sample

Just over half of the students were female (56.5%), and the average age was 18.5 years old. The authors did not provide information about race/ethnicity or financial status, but about half of the students had a non-English mother tongue, and about half (51.8%) were non-Canadian citizens or international students (43%). Just under one-quarter (23.8%) of the students were first-generation college students. All students in the sample were registered for first-year economics classes in the fall of 2015.

Intervention Group

There were two interventions implemented in the study. They share the same base intervention, which was an online, 2.5-hour goal-setting exercise that students completed in the first two weeks of the first semester. Students received credit (2% of their final grade) for this base, online, activity if they completed it in the first two weeks. The authors piloted this intervention in the year preceding the present study and found no discernible effects on students academic outcomes, so they used this base intervention as a comparison condition. The first intervention is the In-Person Coaching ("Online + Coaching"). For this intervention, students completed the online goal setting exercise and were then assigned to an upper-year undergraduate student who served as their personal coach and was available to meet with students to answer any questions (via phone, skype, or in-person). Each coach was assigned to four or five students. About half of the meetings were virtual (skype or text) and half were in-person. Coaches also sent students regular text and email messages offering advice, encouragement and motivation. Coaches provided support and guided students through problems. Topics discussed during meetings included the location of campus services, how to select a major, questions about their academic coursework, getting jobs on campus, their feelings (nervousness, anxiety, sadness) and how to book appointments with counselors. The second intervention was the Text Messaging ("Online + Text Messaging"). For this intervention, students again completed the base online goal setting exercise and were assigned to receive email and text messages at a frequency that they choose (once a week, 2-3 times per week, or 3 or more times per week). They all received email messages, but if they provided their phone number, they received both email and text messages (the same messages). The messages focused on academic and study preparation advice, information on university resources, and motivation and encouragement. The aim of these messages was to provide "nudges" to students. This text messaging campaign was called You@UofT.

Comparison Group

Students in this condition completed a 2.5-hour online goal-setting exercise (as described above). The rest of the academic year, students had access to business as usual services and facilities on their respective campuses.

Support for implementation

Coaches were provided with materials that included the text and email messages to send to students.

 

Your export should download shortly as a zip archive.

This download will include data files for study and findings review data and a data dictionary.

Connect With the WWC

loading
back to top