Digital Messaging to Improve College Enrollment and Success.
Avery, C., Castleman, B.L., Hurwitz, M., Long, B. T., & Page, L. C. (2021). Economics of Education Review, 84, 102170. https://doi.org/10.1016/j.econedurev.2021.102170.
-
examining70,285Students, grades11-PS
Practice Guide
Review Details
Reviewed: July 2024
- Practice Guide (findings for Text-message based outreach and advising to students—Avery et al. (2021))
- Randomized Controlled Trial
- Meets WWC standards without reservations because it is a cluster randomized controlled trial with low cluster-level attrition and individual-level non-response.
This review may not reflect the full body of research evidence for this intervention.
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Findings
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|
|---|---|---|---|---|---|---|---|---|---|
|
Enrolled in any NSC-covered college for one academic semester within 6 months of graduation |
Text-message based outreach and advising to students—Avery et al. (2021) vs. Automated text-message based outreach |
6 Months |
Full sample;
|
59.40 |
60.70 |
Yes |
|
||
| Show Supplemental Findings | |||||||||
|
Enrolled in any 2-year NSC-covered college for one academic semester within 18 months of graduation |
Text-message based outreach and advising to students—Avery et al. (2021) vs. Automated text-message based outreach |
18 Months |
Full sample;
|
18.10 |
18.30 |
No |
-- | ||
|
Enrolled in any NSC-covered college full-time for one academic semester within 6 months of graduation |
Text-message based outreach and advising to students—Avery et al. (2021) vs. Automated text-message based outreach |
6 Months |
Full sample;
|
47.90 |
49.20 |
Yes |
|
||
|
Enrolled in any 4-year NSC-covered college for one academic semester within 6 months of graduation |
Text-message based outreach and advising to students—Avery et al. (2021) vs. Automated text-message based outreach |
6 Months |
Full sample;
|
39.90 |
40.90 |
No |
-- | ||
|
Enrolled in any 2-year NSC-covered college for one academic semester within 6 months of graduation |
Text-message based outreach and advising to students—Avery et al. (2021) vs. Automated text-message based outreach |
6 Months |
Full sample;
|
19.40 |
19.80 |
No |
-- | ||
|
Enrolled in any NSC-covered college for one academic semester within 18 months of graduation |
Text-message based outreach and advising to students—Avery et al. (2021) vs. Automated text-message based outreach |
18 Months |
Full sample;
|
54.40 |
55.50 |
Yes |
|
||
|
Enrolled in any 4-year NSC-covered college for one academic semester within 18 months of graduation |
Text-message based outreach and advising to students—Avery et al. (2021) vs. Automated text-message based outreach |
18 Months |
Full sample;
|
36.30 |
37.20 |
No |
-- | ||
|
Enrolled in any NSC-covered college full-time for one academic semester within 18 months of graduation |
Text-message based outreach and advising to students—Avery et al. (2021) vs. Automated text-message based outreach |
18 Months |
Full sample;
|
41.40 |
42.50 |
Yes |
|
||
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|---|---|---|---|---|---|---|---|---|
|
Took the SAT |
Text-message based outreach and advising to students—Avery et al. (2021) vs. Automated text-message based outreach |
0 Days |
Full sample;
|
66.30 |
67.50 |
No |
-- | |
|
Total number of colleges and scholarship organizations to which students sent SAT scores |
Text-message based outreach and advising to students—Avery et al. (2021) vs. Automated text-message based outreach |
0 Days |
Full sample;
|
1.90 |
2.00 |
No |
-- | |
|
Percentage of grade 12 students who submitted the Free Application for Federal Student Aid (FAFSA®) form |
Text-message based outreach and advising to students—Avery et al. (2021) vs. Automated text-message based outreach |
0 Days |
Full sample (school-level);
|
63.10 |
63.60 |
No |
-- | |
|
Percentage of grade 12 students who completed the Free Application for Federal Student Aid (FAFSA®) form |
Text-message based outreach and advising to students—Avery et al. (2021) vs. Automated text-message based outreach |
0 Days |
Full sample (School-level);
|
57.30 |
57.90 |
No |
-- |
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|---|---|---|---|---|---|---|---|---|
|
Probability of persisting in college for two consecutive fall semesters after the end of grade 12 |
Text-message based outreach and advising to students—Avery et al. (2021) vs. Automated text-message based outreach |
18 Months |
Full sample;
|
48.40 |
49.60 |
Yes |
|
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Sample Characteristics
Characteristics of study sample as reported by study author.
-
Female: 55%
Male: 45% -
Race Asian 7% Black 20% Other or unknown 49% White 24% -
Ethnicity Hispanic 41% Not Hispanic or Latino 59% -
Eligible for Free and Reduced Price Lunch Other or unknown 100%
Study Details
Setting
The national study sample took place in 745 public and private high schools in 15 states across the United States. The Texas study sample included 72 high schools located within eight public school districts in Texas.
Study sample
For the national sample, the researchers randomly assigned approximately one-half of the 745 schools to the intervention group and the remaining schools to the comparison group. A total of 70,285 students in grade 11 were included in the study (the intervention was offered to these students through grade 12). These 70,285 students were enrolled in all 745 randomly assigned schools. Students in the sample were eligible for participation if they took the PSAT during their 11th grade year. Approximately 55% of the students were female, 24% were White, 20% were Black, 7% were Asian, and 49% did not report race. 41% were Hispanic or Latino, and 59% were non-Hispanic or Latino. For the Texas sample, the study included 21,001 students across 72 high schools, 30% of those students were White, 14 % were Black, 6% were Asian, and 3% were Native American. Over half of the study sample (56%) was Hispanic and approximately half was male. Over half of the study sample (55%) was considered economically disadvantaged.
Intervention Group
The intervention is a program that sent scripted text messages to students on a variety of topics related to college readiness and enrollment, such as SAT taking, college applications, financial aid processes, and steps required for enrollment. The text messages were designed to elicit student engagement via text message with a specific full-time college advisor who responded with individualized follow-up communications. Text messages were sent approximately monthly for 15 months – from April of students' 11th grade year through the July following their senior year. Approximately 14 percent of intervention students did not provide a valid contact number, and thus did not receive any text messages. In some schools, the program also included email-based outreach to school counselors with information about their students’ progress with college readiness activities, as well as specific steps they could take to improve these outcomes.
Comparison Group
Students in the comparison group received limited text message outreach, consisting of more general messages approximately every other month. Students who responded received a pre-specified automatic sequence of texts on the same topic of the original text message.
Support for implementation
A partner organization to the study team provided both initial and ongoing training and supervised the college counselors. Trainings covered implementation of the intervention and content related to the intervention, including college affordability, financial aid processes and policies, college entrance exams, and other, similar topics. They also practiced delivering advising over text messages. Prior to each text messages, the partner organization also provided information on the message content, the goal for each message and how to provide counseling related to the topic of the message.
Practice Guide
Review Details
Reviewed: July 2024
- Practice Guide (findings for Text-message based outreach and advising to students—Avery et al. (2021))
- Randomized Controlled Trial
- Meets WWC standards with reservations because it is a cluster randomized controlled trial with low cluster-level attrition that provides evidence of effects on clusters by demonstrating that the analytic sample of individuals is representative of the clusters.
This review may not reflect the full body of research evidence for this intervention.
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Findings
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|
|---|---|---|---|---|---|---|---|---|---|
|
Enrolled in any NSC-covered postsecondary institution within six months of graduation |
Text-message based outreach and advising to students—Avery et al. (2021) vs. Access to text messaging platform |
0 Days |
Full sample;
|
53.00 |
51.30 |
No |
-- | ||
| Show Supplemental Findings | |||||||||
|
Enrolled in any NSC-covered college full-time for one academic semester within 6 months of graduation |
Text-message based outreach and advising to students—Avery et al. (2021) vs. Access to text messaging platform |
0 Days |
Full sample;
|
36.00 |
33.10 |
Yes |
|
||
|
Enrolled in any NSC-covered college for one academic semester within 18 months of graduation |
Text-message based outreach and advising to students—Avery et al. (2021) vs. Access to text messaging platform |
1 Year |
Full sample;
|
49.90 |
47.90 |
No |
-- | ||
|
Enrolled in any NSC-covered college for one academic semester within 18 months of graduation |
Text-message based outreach and advising to students—Avery et al. (2021) vs. Access to text messaging platform |
1 Year |
Students with low (<3.0) GPA and free and reduced-price lunch status;
|
24.50 |
23.30 |
No |
-- | ||
|
Enrolled in any 4-year NSC-covered college for one academic semester within 6 months of graduation |
Text-message based outreach and advising to students—Avery et al. (2021) vs. Access to text messaging platform |
0 Days |
Full sample;
|
33.70 |
32.80 |
No |
-- | ||
|
Enrolled in any 2-year NSC-covered college for one academic semester within 6 months of graduation |
Text-message based outreach and advising to students—Avery et al. (2021) vs. Access to text messaging platform |
0 Days |
Full sample;
|
19.80 |
19.00 |
No |
-- | ||
|
Enrolled in any 4-year NSC-covered college for one academic semester within 18 months of graduation |
Text-message based outreach and advising to students—Avery et al. (2021) vs. Access to text messaging platform |
1 Year |
Full sample;
|
30.60 |
29.70 |
No |
-- | ||
|
Enrolled in any 2-year NSC-covered college for one academic semester within 18 months of graduation |
Text-message based outreach and advising to students—Avery et al. (2021) vs. Access to text messaging platform |
1 Year |
Full sample;
|
19.80 |
18.90 |
No |
-- | ||
|
Enrolled in any NSC-covered college for one academic semester within 6 months of graduation |
Text-message based outreach and advising to students—Avery et al. (2021) vs. Access to text messaging platform |
0 Days |
Students with low (<3.0) GPA and free and reduced-price lunch status;
|
28.00 |
27.00 |
No |
-- | ||
|
College Enrollment Full Time |
Text-message based outreach and advising to students—Avery et al. (2021) vs. Access to text messaging platform |
1 Year |
Full sample;
|
31.70 |
29.30 |
No |
-- | ||
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|
|---|---|---|---|---|---|---|---|---|---|
|
Applied to at least one public, in-state college |
Text-message based outreach and advising to students—Avery et al. (2021) vs. Access to text messaging platform |
0 Days |
Full sample;
|
78.80 |
71.00 |
No |
-- | ||
|
Completed the Free Application for Federal Student Aid (FAFSA®) form |
Text-message based outreach and advising to students—Avery et al. (2021) vs. Access to text messaging platform |
0 Days |
Full sample;
|
49.90 |
45.20 |
No |
-- | ||
|
Took the SAT |
Text-message based outreach and advising to students—Avery et al. (2021) vs. Access to text messaging platform |
0 Days |
Full sample;
|
51.80 |
47.50 |
No |
-- | ||
|
Number of applications submitted to public, in-state colleges |
Text-message based outreach and advising to students—Avery et al. (2021) vs. Access to text messaging platform |
0 Days |
Full sample;
|
2.03 |
1.92 |
No |
-- | ||
|
Percentage of grade 12 students who submitted the Free Application for Federal Student Aid (FAFSA®) form |
Text-message based outreach and advising to students—Avery et al. (2021) vs. Access to text messaging platform |
0 Days |
Full sample (School-level);
|
64.80 |
56.70 |
No |
-- | ||
| Show Supplemental Findings | |||||||||
|
Applied to at least one public, in-state college |
Text-message based outreach and advising to students—Avery et al. (2021) vs. Access to text messaging platform |
0 Days |
Students with low (<3.0) GPA and free and reduced-price lunch status;
|
70.90 |
62.60 |
Yes |
|
||
|
Took the SAT |
Text-message based outreach and advising to students—Avery et al. (2021) vs. Access to text messaging platform |
0 Days |
Students with low (<3.0) GPA and free and reduced-price lunch status;
|
43.00 |
37.10 |
Yes |
|
||
|
Completed the Free Application for Federal Student Aid (FAFSA®) form |
Text-message based outreach and advising to students—Avery et al. (2021) vs. Access to text messaging platform |
0 Days |
Students with low (<3.0) GPA and free and reduced-price lunch status;
|
35.20 |
33.40 |
No |
-- | ||
|
Percentage of grade 12 students who completed the Free Application for Federal Student Aid (FAFSA®) form |
Text-message based outreach and advising to students—Avery et al. (2021) vs. Access to text messaging platform |
0 Days |
Full sample (School-level) ;
|
61.50 |
52.00 |
-- |
-- | ||
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|
|---|---|---|---|---|---|---|---|---|---|
|
Enrolled in any NSC-covered college for two consecutive fall semesters after the end of grade 12 |
Text-message based outreach and advising to students—Avery et al. (2021) vs. Access to text messaging platform |
1 Year |
Full sample;
|
48.40 |
47.50 |
No |
-- | ||
| Show Supplemental Findings | |||||||||
|
Enrolled in any NSC-covered college for two consecutive fall semesters after the end of grade 12 |
Text-message based outreach and advising to students—Avery et al. (2021) vs. Access to text messaging platform |
1 Year |
Students with low (<3.0) GPA and free and reduced-price lunch status;
|
22.70 |
22.80 |
No |
-- | ||
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Sample Characteristics
Characteristics of study sample as reported by study author.
-
Female: 50%
Male: 50% -
- B
- A
- C
- D
- E
- F
- G
- I
- H
- J
- K
- L
- P
- M
- N
- O
- Q
- R
- S
- V
- U
- T
- W
- X
- Z
- Y
- a
- h
- i
- b
- d
- e
- f
- c
- g
- j
- k
- l
- m
- n
- o
- p
- q
- r
- s
- t
- u
- v
- x
- w
- y
Texas
-
Race Asian 6% Black 14% Native American 3% Other or unknown 47% White 31% -
Ethnicity Hispanic 56% Not Hispanic or Latino 44% -
Eligible for Free and Reduced Price Lunch Free or reduced price lunch (FRPL) 55% No FRPL 45%
Study Details
Setting
The national study sample took place in 745 public and private high schools in 15 states across the United States. The Texas study sample included 72 high schools located within eight public school districts in Texas.
Study sample
For the national sample, the researchers randomly assigned approximately one-half of the 745 schools to the intervention group and the remaining schools to the comparison group. A total of 70,285 students in grade 11 were included in the study (the intervention was offered to these students through grade 12). These 70,285 students were enrolled in all 745 randomly assigned schools. Students in the sample were eligible for participation if they took the PSAT during their 11th grade year. Approximately 55% of the students were female, 24% were White, 20% were Black, 7% were Asian, and 49% did not report race. 41% were Hispanic or Latino, and 59% were non-Hispanic or Latino. For the Texas sample, the study included 21,001 students, 30% of those students were White, 14 % were Black, 6% were Asian, and 3% were Native American. Over half of the study sample (56%) was Hispanic and approximately half was male. Over half of the study sample (55%) was considered economically disadvantaged.
Intervention Group
The intervention consisted of a text message-based outreach program, provided to individual students, to support them in specific steps in college enrollment and success. Students received text-message based reminders every one to two weeks from their school’s college counselor about college-going tasks and related deadlines, and were also invited to follow-up with their counselor at school for additional guidance and supports. Counselors could customize the text messages based on students' completion of steps in the college-going process, and school districts could decide which time of day text messages were sent. Students received texts and additional advising supports from the spring of grade 11 through the summer after high school graduation, which was designed to cover the college search, application, selection, and transition processes.
Comparison Group
Schools in the comparison condition had access to the online student information system from OneLogos Education Solutions that supported text communication between students and counselors; however, their schools rarely used it to provide information to students or send reminders about key steps in the college application process.
Support for implementation
The study authors provided draft messages to school counselors who sent the text messages to students, but school districts had the discretion to adjust the content and timing of their messages based on district context.
An indicator of the effect of the intervention, the improvement index can be interpreted as the expected change in percentile rank for an average comparison group student if that student had received the intervention.
For more, please see the WWC Glossary entry for improvement index.
An outcome is the knowledge, skills, and attitudes that are attained as a result of an activity. An outcome measures is an instrument, device, or method that provides data on the outcome.
A finding that is included in the effectiveness rating. Excluded findings may include subgroups and subscales.
The sample on which the analysis was conducted.
The group to which the intervention group is compared, which may include a different intervention, business as usual, or no services.
The timing of the post-intervention outcome measure.
The number of students included in the analysis.
The mean score of students in the intervention group.
The mean score of students in the comparison group.
The WWC considers a finding to be statistically significant if the likelihood that the finding is due to chance alone, rather than a real difference, is less than five percent.
The WWC reviews studies for WWC products, Department of Education grant competitions, and IES performance measures.
The name and version of the document used to guide the review of the study.
The version of the WWC design standards used to guide the review of the study.
The result of the WWC assessment of the study. The rating is based on the strength of evidence of the effectiveness of the intervention. Studies are given a rating of Meets WWC Design Standards without Reservations, Meets WWC Design Standards with Reservations, or >Does Not Meet WWC Design Standards.
A related publication that was reviewed alongside the main study of interest.
Study findings for this report.
Based on the direction, magnitude, and statistical significance of the findings within a domain, the WWC characterizes the findings from a study as one of the following: statistically significant positive effects, substantively important positive effects, indeterminate effects, substantively important negative effects, and statistically significant negative effects. For more, please see the WWC Handbook.
The WWC may review studies for multiple purposes, including different reports and re-reviews using updated standards. Each WWC review of this study is listed in the dropdown. Details on any review may be accessed by making a selection from the drop down list.
Tier 1 Strong indicates strong evidence of effectiveness,
Tier 2 Moderate indicates moderate evidence of effectiveness, and
Tier 3 Promising indicates promising evidence of effectiveness,
as defined in the
non-regulatory guidance for ESSA
and the regulations for ED discretionary grants (EDGAR Part 77).