Varying Levels of Success
Sommo, Colleen; Slaughter, Austin; Saunier, Cyrette; Scrivener, Susan; Warner, Kayla (2023). MDRC. Retrieved from: https://eric.ed.gov/?id=ED631888
-
examining2,972Students, gradePS
Single Study Review
Review Details
Reviewed: October 2024
- Single Study Review (findings for Scaling Up College Completion Efforts for Student Success (SUCCESS))
- Randomized Controlled Trial
- Meets WWC standards without reservations because it is a randomized controlled trial with low attrition.
This review may not reflect the full body of research evidence for this intervention.
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Findings
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
|---|---|---|---|---|---|---|---|
|
Student earned a credential within 2 semesters (associate degree or certificate) |
Scaling Up College Completion Efforts for Student Success (SUCCESS) vs. Business as usual |
-4 Semesters |
Full sample;
|
7.00 |
8.00 |
No |
-- |
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
||
|---|---|---|---|---|---|---|---|---|---|
|
Student enrolled full time in 3rd semester of college |
Scaling Up College Completion Efforts for Student Success (SUCCESS) vs. Business as usual |
-3 Semesters |
Full sample;
|
35.20 |
32.00 |
No |
-- | ||
|
Student enrolled in 3rd semester of college |
Scaling Up College Completion Efforts for Student Success (SUCCESS) vs. Business as usual |
-3 Semesters |
Full sample;
|
55.30 |
54.00 |
No |
-- | ||
| Show Supplemental Findings | |||||||||
|
Student enrolled full time in 1st semester of college |
Scaling Up College Completion Efforts for Student Success (SUCCESS) vs. Business as usual |
-5 Semesters |
Full sample;
|
66.90 |
63.10 |
Yes |
|
||
|
Student enrolled in 1st semester of college |
Scaling Up College Completion Efforts for Student Success (SUCCESS) vs. Business as usual |
-5 Semesters |
Full sample;
|
89.30 |
88.40 |
No |
-- | ||
|
Student enrolled full time in 2nd semester of college |
Scaling Up College Completion Efforts for Student Success (SUCCESS) vs. Business as usual |
-4 Semesters |
Full sample;
|
52.80 |
50.40 |
No |
-- | ||
|
Student enrolled in 2nd semester of college |
Scaling Up College Completion Efforts for Student Success (SUCCESS) vs. Business as usual |
-4 Semesters |
Full sample;
|
72.90 |
71.80 |
No |
-- | ||
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
||
|---|---|---|---|---|---|---|---|---|---|
|
Credits earned in 2 semesters of college |
Scaling Up College Completion Efforts for Student Success (SUCCESS) vs. Business as usual |
-4 Semesters |
Full sample;
|
15.79 |
15.87 |
No |
-- | ||
| Show Supplemental Findings | |||||||||
|
Credits attempted in 1 semester of college |
Scaling Up College Completion Efforts for Student Success (SUCCESS) vs. Business as usual |
-5 Semesters |
Full sample;
|
11.54 |
11.09 |
Yes |
|
||
|
Credits attempted in 2 semesters of college |
Scaling Up College Completion Efforts for Student Success (SUCCESS) vs. Business as usual |
-4 Semesters |
Full sample;
|
21.77 |
21.19 |
No |
-- | ||
|
Credits attempted in 3 semesters of college |
Scaling Up College Completion Efforts for Student Success (SUCCESS) vs. Business as usual |
-3 Semesters |
Full sample;
|
28.00 |
27.31 |
No |
-- | ||
|
Credits earned in 1 semester of college |
Scaling Up College Completion Efforts for Student Success (SUCCESS) vs. Business as usual |
-5 Semesters |
Full sample;
|
8.29 |
8.18 |
No |
-- | ||
|
Credits earned in 2 semesters of college |
Scaling Up College Completion Efforts for Student Success (SUCCESS) vs. Business as usual |
-4 Semesters |
Black or African American;
|
13.78 |
13.49 |
No |
-- | ||
|
Cumulative Total credits earned |
Scaling Up College Completion Efforts for Student Success (SUCCESS) vs. Business as usual |
-4 Semesters |
White;
|
14.74 |
14.98 |
No |
-- | ||
|
Credits earned in 2 semesters of college |
Scaling Up College Completion Efforts for Student Success (SUCCESS) vs. Business as usual |
-4 Semesters |
Hispanic or Latino;
|
17.65 |
18.44 |
No |
-- | ||
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Sample Characteristics
Characteristics of study sample as reported by study author.
-
Female: 68%
Male: 31%
Other or unknown: 1% -
Rural, Suburban, Urban
-
- B
- A
- C
- D
- E
- F
- G
- I
- H
- J
- K
- L
- P
- M
- N
- O
- Q
- R
- S
- V
- U
- T
- W
- X
- Z
- Y
- a
- h
- i
- b
- d
- e
- f
- c
- g
- j
- k
- l
- m
- n
- o
- p
- q
- r
- s
- t
- u
- v
- x
- w
- y
California, Indiana, New Jersey, Ohio
-
Race Asian 4% Black 28% Native American 0% Other or unknown 40% Two or more races 3% White 25% -
Ethnicity Hispanic 38% Not Hispanic or Latino 61% Other or unknown 1%
Study Details
Setting
The study took place in 7 two- and four-year colleges in rural, suburban, and urban areas of California, Indiana, New Jersey, and Ohio. Participating colleges included: Bakersfield College in California; Ivy Tech Bloomington, Ivy Tech Indianapolis, and Ivy Tech Kokomo in Indiana; Essex County College and Passaic County Community College in New Jersey; and Stark State College in Ohio.
Study sample
The researchers randomly assigned 1,704 students to the intervention group and 1,285 students to the comparison group. The study targeted degree- or certificate-seeking students in their first year of college who were willing to enroll in school full time. The main analytic sample has 1,693 students in the intervention group and 1,279 students in the comparison group. Among students eligible for the study, 31% were male and 68% were female. Approximately 28% were Black, 25% were White, 4% were Asian or Pacific Islander, fewer than 1% were Native American, 3% were more than one race, and 39% did not report race or were of another race. Thirty-eight percent were Hispanic or Latino. Two-thirds of the students were 23 years old or younger and approximately half (52 percent) were employed. Sample characteristics varied widely across participating colleges; for example, the proportion of Black students varies from 4% to 73% across colleges; the proportion of Hispanic students ranges from 4% to 93%; and the proportion of White students ranges from less than 1% to 78%.
Intervention Group
Students in the intervention group participated in Scaling Up College Completion Efforts for Student Success (SUCCESS), a program designed to promote students’ educational and financial success. Through SUCCESS, students meet with coaches at least once a month to discuss personal or academic issues. In the first semester, students are required to meet with their coaches two times per month; students struggling academically or experiencing personal issues are also required to meet two times per month beyond the first semester. Students are typically required to enroll in college full-time (in at least 24 credits per year) and are encouraged to enroll in summer courses. Students who meet with their coaches regularly and satisfy the credit enrollment requirements receive $50 per month. SUCCESS staff collect data to track students’ participation and progress as well as for program management and improvement planning purposes. The researchers highlight key changes in the SUCCESS programs that took place during the study period due to the COVID-19 pandemic. For example, by fall 2020, the SUCCESS programs provided virtual coaching and did not strictly enforce the full-time enrollment requirement, allowing part-time students to participate in SUCCESS and receive incentives. The intervention is intended to last 3 years, over a period of 6 semesters. In this study, the researchers measured outcomes after the first, second, and third semester of college, which corresponds to 5th, 4th, and 3rd semesters before the end of the intervention.
Comparison Group
Students in the comparison group were offered standard school services.
Support for implementation
The study did not describe any support or training offered to the providers of the intervention.
Additional Sources
In the case of multiple manuscripts that report on one study, the WWC selects one manuscript as the primary citation and lists other manuscripts that describe the study as additional sources.
-
Sommo, Colleen; Lepe, Marco; Ratledge, Alyssa. (2022). Supporting College Students during the Pandemic: Early Lessons from SUCCESS. MDRC.
An indicator of the effect of the intervention, the improvement index can be interpreted as the expected change in percentile rank for an average comparison group student if that student had received the intervention.
For more, please see the WWC Glossary entry for improvement index.
An outcome is the knowledge, skills, and attitudes that are attained as a result of an activity. An outcome measures is an instrument, device, or method that provides data on the outcome.
A finding that is included in the effectiveness rating. Excluded findings may include subgroups and subscales.
The sample on which the analysis was conducted.
The group to which the intervention group is compared, which may include a different intervention, business as usual, or no services.
The timing of the post-intervention outcome measure.
The number of students included in the analysis.
The mean score of students in the intervention group.
The mean score of students in the comparison group.
The WWC considers a finding to be statistically significant if the likelihood that the finding is due to chance alone, rather than a real difference, is less than five percent.
The WWC reviews studies for WWC products, Department of Education grant competitions, and IES performance measures.
The name and version of the document used to guide the review of the study.
The version of the WWC design standards used to guide the review of the study.
The result of the WWC assessment of the study. The rating is based on the strength of evidence of the effectiveness of the intervention. Studies are given a rating of Meets WWC Design Standards without Reservations, Meets WWC Design Standards with Reservations, or >Does Not Meet WWC Design Standards.
A related publication that was reviewed alongside the main study of interest.
Study findings for this report.
Based on the direction, magnitude, statistical significance, and sample size of the findings within a domain, the WWC assigns effectiveness ratings as one of the following: Tier 1 (strong evidence), Tier 2 (moderate evidence), Tier 3 (promising evidence), uncertain effects, and negative effects. For more detail, please see the WWC Handbook.
The WWC may review studies for multiple purposes, including different reports and re-reviews using updated standards. Each WWC review of this study is listed in the dropdown. Details on any review may be accessed by making a selection from the drop down list.
Tier 1 Strong indicates strong evidence of effectiveness,
Tier 2 Moderate indicates moderate evidence of effectiveness, and
Tier 3 Promising indicates promising evidence of effectiveness,
as defined in the
non-regulatory guidance for ESSA
and the regulations for ED discretionary grants (EDGAR Part 77).