Does virtual advising increase college enrollment? Evidence from a random-assignment college access field experiment. [V-SOURCE Milestones vs. business as usual control]
Phillips, M., & Reber, S. (2022). American Economic Journal: Economic Policy, 14(3), 198-234.
-
examining5,089Students, grades11-12
Practice Guide
Review Details
Reviewed: August 2023
- Practice Guide (findings for Virtual Student Outreach for College Enrollment (V-SOURCE) Milestones)
- Randomized Controlled Trial
- Meets WWC standards without reservations because it is a randomized controlled trial with low attrition.
This review may not reflect the full body of research evidence for this intervention.
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Findings
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|
|---|---|---|---|---|---|---|---|---|---|
|
Enrolled in any college |
Virtual Student Outreach for College Enrollment (V-SOURCE) Milestones vs. Business as usual |
3 Months |
Full sample;
|
71.00 |
70.50 |
No |
-- | ||
|
Accepted to any 4 year college, self-report |
Virtual Student Outreach for College Enrollment (V-SOURCE) Milestones vs. Business as usual |
0 Days |
Full sample;
|
67.30 |
67.30 |
No |
-- | ||
| Show Supplemental Findings | |||||||||
|
Enrolled in any college in University of California system |
Virtual Student Outreach for College Enrollment (V-SOURCE) Milestones vs. Business as usual |
3 Months |
Hispanic or Latino, Spanish speaking at home;
|
13.60 |
10.40 |
Yes |
|
||
|
Enrolled in any selective 4 year college |
Virtual Student Outreach for College Enrollment (V-SOURCE) Milestones vs. Business as usual |
3 Months |
Hispanic or Latino, Spanish speaking at home;
|
11.50 |
9.40 |
No |
-- | ||
|
Enrolled in any selective 4 year college |
Virtual Student Outreach for College Enrollment (V-SOURCE) Milestones vs. Business as usual |
3 Months |
Full sample;
|
13.30 |
11.70 |
No |
-- | ||
|
Enrolled in any college in University of California system |
Virtual Student Outreach for College Enrollment (V-SOURCE) Milestones vs. Business as usual |
3 Months |
Full sample;
|
14.30 |
12.70 |
No |
-- | ||
|
Accepted to any college in University of California system, self-report |
Virtual Student Outreach for College Enrollment (V-SOURCE) Milestones vs. Business as usual |
0 Days |
Hispanic or Latino, Spanish speaking at home;
|
29.40 |
26.60 |
No |
-- | ||
|
Enrolled in any college |
Virtual Student Outreach for College Enrollment (V-SOURCE) Milestones vs. Business as usual |
3 Months |
Hispanic or Latino, Spanish speaking at home;
|
68.40 |
65.80 |
No |
-- | ||
|
Enrolled in any 4-year college |
Virtual Student Outreach for College Enrollment (V-SOURCE) Milestones vs. Business as usual |
3 Months |
Hispanic or Latino, Spanish speaking at home;
|
41.80 |
39.30 |
No |
-- | ||
|
Accepted to any 4 year college, self-report |
Virtual Student Outreach for College Enrollment (V-SOURCE) Milestones vs. Business as usual |
0 Days |
Hispanic or Latino, Spanish speaking at home;
|
67.70 |
66.20 |
No |
-- | ||
|
Accepted to any selective college, self-report |
Virtual Student Outreach for College Enrollment (V-SOURCE) Milestones vs. Business as usual |
0 Days |
Hispanic or Latino, Spanish speaking at home;
|
22.40 |
21.00 |
No |
-- | ||
|
Accepted to any selective college, self-report |
Virtual Student Outreach for College Enrollment (V-SOURCE) Milestones vs. Business as usual |
0 Days |
Full sample;
|
23.90 |
23.40 |
No |
-- | ||
|
Accepted to any college in University of California system, self-report |
Virtual Student Outreach for College Enrollment (V-SOURCE) Milestones vs. Business as usual |
0 Days |
Full sample;
|
30.50 |
29.50 |
No |
-- | ||
|
Accepted to any college in California State University system, self-report |
Virtual Student Outreach for College Enrollment (V-SOURCE) Milestones vs. Business as usual |
0 Days |
Hispanic or Latino, Spanish speaking at home;
|
64.10 |
62.70 |
No |
-- | ||
|
Enrolled in any 4-year college |
Virtual Student Outreach for College Enrollment (V-SOURCE) Milestones vs. Business as usual |
3 Months |
Full sample;
|
43.60 |
43.30 |
No |
-- | ||
|
Accepted to any college in California State University system, self-report |
Virtual Student Outreach for College Enrollment (V-SOURCE) Milestones vs. Business as usual |
0 Days |
Full sample;
|
62.00 |
61.60 |
No |
-- | ||
|
Enrolled in any college in California State University system |
Virtual Student Outreach for College Enrollment (V-SOURCE) Milestones vs. Business as usual |
3 Months |
Full sample;
|
23.60 |
24.90 |
No |
-- | ||
|
Enrolled in any college in California State University system |
Virtual Student Outreach for College Enrollment (V-SOURCE) Milestones vs. Business as usual |
3 Months |
Hispanic or Latino, Spanish speaking at home;
|
24.10 |
26.00 |
No |
-- | ||
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|
|---|---|---|---|---|---|---|---|---|---|
|
Completed the FAFSA |
Virtual Student Outreach for College Enrollment (V-SOURCE) Milestones vs. Business as usual |
0 Days |
Full sample;
|
81.70 |
78.90 |
No |
-- | ||
|
Applied to any 4-year college, self report |
Virtual Student Outreach for College Enrollment (V-SOURCE) Milestones vs. Business as usual |
0 Days |
Full sample;
|
80.40 |
77.90 |
No |
-- | ||
|
Took SAT/ACT, self-report |
Virtual Student Outreach for College Enrollment (V-SOURCE) Milestones vs. Business as usual |
0 Days |
Full sample;
|
84.60 |
82.90 |
No |
-- | ||
|
Submitted college applications to at least two systems, self-report |
Virtual Student Outreach for College Enrollment (V-SOURCE) Milestones vs. Business as usual |
0 Days |
Full sample;
|
49.50 |
48.90 |
No |
-- | ||
|
Sought information about college application process, self-report (researcher-developed scale) |
Virtual Student Outreach for College Enrollment (V-SOURCE) Milestones vs. Business as usual |
0 Days |
Full sample;
|
-0.03 |
0.00 |
No |
-- | ||
| Show Supplemental Findings | |||||||||
|
Completed the FAFSA |
Virtual Student Outreach for College Enrollment (V-SOURCE) Milestones vs. Business as usual |
0 Days |
Hispanic or Latino, Spanish speaking at home;
|
84.80 |
81.30 |
Yes |
|
||
|
Applied to any 4-year college, self report |
Virtual Student Outreach for College Enrollment (V-SOURCE) Milestones vs. Business as usual |
0 Days |
Hispanic or Latino, Spanish speaking at home;
|
80.80 |
77.70 |
Yes |
|
||
|
Took SAT/ACT, self-report |
Virtual Student Outreach for College Enrollment (V-SOURCE) Milestones vs. Business as usual |
0 Days |
Hispanic or Latino, Spanish speaking at home;
|
84.90 |
82.30 |
Yes |
|
||
|
Applied to any college in California State University system, self-report |
Virtual Student Outreach for College Enrollment (V-SOURCE) Milestones vs. Business as usual |
0 Days |
Hispanic or Latino, Spanish speaking at home;
|
77.30 |
74.10 |
Yes |
|
||
|
Registered for the SAT/ACT, self-report |
Virtual Student Outreach for College Enrollment (V-SOURCE) Milestones vs. Business as usual |
0 Days |
Hispanic or Latino, Spanish speaking at home;
|
86.20 |
84.10 |
No |
-- | ||
|
Applied to any college in California State University system, self-report |
Virtual Student Outreach for College Enrollment (V-SOURCE) Milestones vs. Business as usual |
0 Days |
Full sample;
|
75.10 |
72.70 |
No |
-- | ||
|
Applied to any college in University of California system, self-report |
Virtual Student Outreach for College Enrollment (V-SOURCE) Milestones vs. Business as usual |
0 Days |
Hispanic or Latino, Spanish speaking at home;
|
44.40 |
41.80 |
No |
-- | ||
|
Registered for the SAT/ACT, self-report |
Virtual Student Outreach for College Enrollment (V-SOURCE) Milestones vs. Business as usual |
0 Days |
Full sample;
|
86.00 |
84.20 |
No |
-- | ||
|
Applied to any selective 4-year college, self report |
Virtual Student Outreach for College Enrollment (V-SOURCE) Milestones vs. Business as usual |
0 Days |
Hispanic or Latino, Spanish speaking at home;
|
46.40 |
44.80 |
No |
-- | ||
|
Applied to any college in University of California system, self-report |
Virtual Student Outreach for College Enrollment (V-SOURCE) Milestones vs. Business as usual |
0 Days |
Full sample;
|
45.40 |
44.50 |
No |
-- | ||
|
Submitted college applications to at least two systems, self-report |
Virtual Student Outreach for College Enrollment (V-SOURCE) Milestones vs. Business as usual |
0 Days |
Hispanic or Latino, Spanish speaking at home;
|
47.00 |
45.80 |
No |
-- | ||
|
Applied to any selective 4-year college, self report |
Virtual Student Outreach for College Enrollment (V-SOURCE) Milestones vs. Business as usual |
0 Days |
Full sample;
|
47.60 |
47.60 |
No |
-- | ||
|
Sought information about college application process, self-report (researcher-developed scale) |
Virtual Student Outreach for College Enrollment (V-SOURCE) Milestones vs. Business as usual |
0 Days |
Hispanic or Latino, Spanish speaking at home;
|
-0.04 |
-0.04 |
No |
-- | ||
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|
|---|---|---|---|---|---|---|---|---|---|
|
Persisting in college - Any college |
Virtual Student Outreach for College Enrollment (V-SOURCE) Milestones vs. Business as usual |
15 Months |
Full sample;
|
64.10 |
63.30 |
No |
-- | ||
| Show Supplemental Findings | |||||||||
|
Persisting in any college in University of California system |
Virtual Student Outreach for College Enrollment (V-SOURCE) Milestones vs. Business as usual |
15 Months |
Hispanic or Latino, Spanish speaking at home;
|
12.30 |
8.90 |
Yes |
|
||
|
Persisting in college - any selective 4 year college |
Virtual Student Outreach for College Enrollment (V-SOURCE) Milestones vs. Business as usual |
15 Months |
Hispanic or Latino, Spanish speaking at home;
|
10.80 |
8.30 |
Yes |
|
||
|
Persisting in any college in University of California system |
Virtual Student Outreach for College Enrollment (V-SOURCE) Milestones vs. Business as usual |
15 Months |
Full sample;
|
13.10 |
11.50 |
No |
-- | ||
|
Persisting in college - any selective 4 year college |
Virtual Student Outreach for College Enrollment (V-SOURCE) Milestones vs. Business as usual |
15 Months |
Full sample;
|
12.10 |
10.80 |
No |
-- | ||
|
Persisting in college - Any 4 year college |
Virtual Student Outreach for College Enrollment (V-SOURCE) Milestones vs. Business as usual |
15 Months |
Hispanic or Latino, Spanish speaking at home;
|
35.90 |
33.30 |
No |
-- | ||
|
Persisting in college - Any college |
Virtual Student Outreach for College Enrollment (V-SOURCE) Milestones vs. Business as usual |
15 Months |
Hispanic or Latino, Spanish speaking at home;
|
61.50 |
59.50 |
No |
-- | ||
|
Persisting in college - Any 4 year college |
Virtual Student Outreach for College Enrollment (V-SOURCE) Milestones vs. Business as usual |
15 Months |
Full sample;
|
37.10 |
36.70 |
No |
-- | ||
|
Persisting in any college in California State University system |
Virtual Student Outreach for College Enrollment (V-SOURCE) Milestones vs. Business as usual |
15 Months |
Full sample;
|
19.30 |
20.30 |
No |
-- | ||
|
Persisting in any college in California State University system |
Virtual Student Outreach for College Enrollment (V-SOURCE) Milestones vs. Business as usual |
15 Months |
Hispanic or Latino, Spanish speaking at home;
|
20.00 |
21.90 |
No |
-- | ||
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Sample Characteristics
Characteristics of study sample as reported by study author.
-
Female: 68%
Male: 32% -
Rural, Suburban, Town, Urban
-
- B
- A
- C
- D
- E
- F
- G
- I
- H
- J
- K
- L
- P
- M
- N
- O
- Q
- R
- S
- V
- U
- T
- W
- X
- Z
- Y
- a
- h
- i
- b
- d
- e
- f
- c
- g
- j
- k
- l
- m
- n
- o
- p
- q
- r
- s
- t
- u
- v
- x
- w
- y
California
-
Race Asian 11% Black 6% Other or unknown 76% Two or more races 3% White 5% -
Ethnicity Hispanic 76% Not Hispanic or Latino 24% -
Eligible for Free and Reduced Price Lunch Free or reduced price lunch (FRPL) 53% No FRPL 47%
Study Details
Setting
The study took place in 84 high schools in six counties in central and southern California. The majority of schools were in a large city (59%). The remaining schools were in a suburb (27%), small or midsize city (6%), or small town/rural area (9%).
Study sample
Students were randomly assigned to one of three groups: the V-SOURCE Complete intervention group, the V-SOURCE Milestones intervention group, and the comparison group. This review focuses on the V-SOURCE Milestones and comparison groups. The researchers randomly assigned 2,571 students to the V-SOURCE Milestones intervention group and 2,577 students to the comparison group. A total of 5,089 students in grades 11 and 12 were included in the study. The students attended 84 high schools in California where a majority of students were economically disadvantaged. Approximately 68% of the sample was female, and 53% were eligible for free or reduced-price lunch. Seventy-six percent did not report race, 11% were Asian, 6% were Black, and 5% were White. Seventy-six percent were Hispanic or Latino, and 24% were non-Hispanic or Latino. About 52% spoke Spanish at home.
Intervention Group
V-SOURCE is an intervention designed to increase college enrollment among economically disadvantaged students on track to be eligible for admission to a public, 4-year college. It is an adaptation of the Student Outreach for College Enrollment (SOURCE) program, which provides in-person, peer advisors for high school students to support them in applying and enrolling in college. The V-SOURCE Milestones intervention includes several components: (1) access to a website with information and guidance on the college application process, (2) access to an SAT preparation curriculum, (3) weekly reminders for specific steps in the college and financial aid application process, and (4) incentives for completing milestones. The V-SOURCE website includes information on college and financial aid applications and allows students to track their individual progress on SAT prep and on college applications. The SAT preparation curriculum, called Ready, SAT, Go!, is available online for students to access, if they choose to do so. This curriculum is intended to assist lower-scoring students prepare for the SAT. Students receive automated electronic reminders via email and text about approaching deadlines, such as the deadline for SAT registration, college applications, or financial aid applications. Students received $20 gift cards for completing each of the four following milestones in the college application process: registering for the SAT, taking the SAT, submitting two college applications to different college systems, and completing the FAFSA. In this study, the V-SOURCE Milestone intervention was implemented starting in March of students’ junior year and continued through June following their high school graduation. The study also provided additional services in the summer after their senior year to a random subset of the intervention group. One-third received automated messages regarding tasks they had to complete during that summer while another third received one message at the beginning of the summer with all the tasks to complete for the summer. (The other one-third of the intervention group received no additional services in the summer but continued to have access to the V-SOURCE website.) A separate study review under this same manuscript summarizes findings for the V-SOURCE Complete study arm. V-SOURCE Complete is a similar intervention to V-SOURCE Milestones, but Complete participants also had access to advisors that provided personalized support.
Comparison Group
Students in the comparison group received business-as-usual counseling and support for preparing college applications.
Support for implementation
EdBoost implemented the V-SOURCE Milestones program. More information on EdBoost is available online at https://www.edboost.org/.
Additional Sources
In the case of multiple manuscripts that report on one study, the WWC selects one manuscript as the primary citation and lists other manuscripts that describe the study as additional sources.
-
Phillips, M., & Reber, S. (2019). Report on the implementation and impacts of the V-SOURCE college access program. UCLA. Retrieved from https://escholarship.org/uc/item/55q9n7ws.
-
Phillips, M., & Reber, S. (2018). When "low touch" is not enough: Evidence from a random assignment college access field experiment. [V-SOURCE Milestones vs. business as usual control]. California Center for Population Research, University of California Los Angeles, Population Working Paper No. PWP-CCPR-2018-008. http://papers.ccpr.ucla.edu/index.php/pwp/article/view/1213/596.
An indicator of the effect of the intervention, the improvement index can be interpreted as the expected change in percentile rank for an average comparison group student if that student had received the intervention.
For more, please see the WWC Glossary entry for improvement index.
An outcome is the knowledge, skills, and attitudes that are attained as a result of an activity. An outcome measures is an instrument, device, or method that provides data on the outcome.
A finding that is included in the effectiveness rating. Excluded findings may include subgroups and subscales.
The sample on which the analysis was conducted.
The group to which the intervention group is compared, which may include a different intervention, business as usual, or no services.
The timing of the post-intervention outcome measure.
The number of students included in the analysis.
The mean score of students in the intervention group.
The mean score of students in the comparison group.
The WWC considers a finding to be statistically significant if the likelihood that the finding is due to chance alone, rather than a real difference, is less than five percent.
The WWC reviews studies for WWC products, Department of Education grant competitions, and IES performance measures.
The name and version of the document used to guide the review of the study.
The version of the WWC design standards used to guide the review of the study.
The result of the WWC assessment of the study. The rating is based on the strength of evidence of the effectiveness of the intervention. Studies are given a rating of Meets WWC Design Standards without Reservations, Meets WWC Design Standards with Reservations, or >Does Not Meet WWC Design Standards.
A related publication that was reviewed alongside the main study of interest.
Study findings for this report.
Based on the direction, magnitude, and statistical significance of the findings within a domain, the WWC characterizes the findings from a study as one of the following: statistically significant positive effects, substantively important positive effects, indeterminate effects, substantively important negative effects, and statistically significant negative effects. For more, please see the WWC Handbook.
The WWC may review studies for multiple purposes, including different reports and re-reviews using updated standards. Each WWC review of this study is listed in the dropdown. Details on any review may be accessed by making a selection from the drop down list.
Tier 1 Strong indicates strong evidence of effectiveness,
Tier 2 Moderate indicates moderate evidence of effectiveness, and
Tier 3 Promising indicates promising evidence of effectiveness,
as defined in the
non-regulatory guidance for ESSA
and the regulations for ED discretionary grants (EDGAR Part 77).