Moving Forward: Early Findings from the Performance-Based Scholarship Demonstration in Arizona. The Performance-Based Scholarship Demonstration
Patel, Reshma; Valenzuela, Ireri (2013). MDRC. Retrieved from: https://eric.ed.gov/?id=ED545467
-
examining1,028Students, gradePS
Strategies for Postsecondary Students in Developmental Education–A Practice Guide for College and University Administrators, Advisors, and Faculty
Review Details
Reviewed: February 2024
Strategies for Postsecondary Students in Developmental Education–A Practice Guide for College and University Administrators, Advisors, and Faculty Practice Guide- Randomized Controlled Trial
- Meets WWC standards without reservations because it is a randomized controlled trial with low attrition.
This review may not reflect the full body of research evidence for this intervention.
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Findings
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|---|---|---|---|---|---|---|---|---|
|
Cumulative GPA (2.0 and higher) through first two semesters |
Adelante vs. Business as usual |
2 Semesters |
Full sample;
|
68.20 |
67.00 |
No |
-- |
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|---|---|---|---|---|---|---|---|---|
|
College enrollment: First semester |
Adelante vs. Business as usual |
1 Semester |
Full sample;
|
98.20 |
97.10 |
No |
-- |
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|---|---|---|---|---|---|---|---|---|
|
Three year graduation rate |
Adelante vs. Business as usual |
3 Years |
Full sample;
|
21.20 |
19.79 |
No |
-- |
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|---|---|---|---|---|---|---|---|---|
|
College-level credits earned after 3 years |
Adelante vs. Business as usual |
3 Years |
Full sample;
|
28.33 |
25.55 |
No |
-- |
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Sample Characteristics
Characteristics of study sample as reported by study author.
-
Male: 100% -
Urban
-
- B
- A
- C
- D
- E
- F
- G
- I
- H
- J
- K
- L
- P
- M
- N
- O
- Q
- R
- S
- V
- U
- T
- W
- X
- Z
- Y
- a
- h
- i
- b
- d
- e
- f
- c
- g
- j
- k
- l
- m
- n
- o
- p
- q
- r
- s
- t
- u
- v
- x
- w
- y
Arizona
-
Race Other or unknown 100% -
Ethnicity Hispanic 100%
Study Details
Setting
The Adelante program study was implemented with low-income, Latino men at Pima Community College, which is a two-year, Hispanic-serving institution with six campus locations throughout the greater Tucson Metropolitan Area in Pima County in southern Arizona. The study had three cohorts (fall 2010, fall 2011, and fall 2012) and tracked outcomes for two semesters for all three cohorts. Pima drew participants from six campuses (Downtown, West, Desert Vista, Northwest, East, and Community), p. 16.
Study sample
100% of the study sample are low-income, Latino males. Three-fourths of the sample are aged 17-26 years old, and one-fourth are 27 years old or older. More than one-third (36.7%) are first-generation to attend college. About six out of ten (59.2%) of the full sample regularly speaks a language other than English at home.
Intervention Group
The Adelante program is a three-semester program, whereby all students receiving the intervention were eligible for awards up to $1,500 per semester (p. 19). The award was broken into three payments, and the first two payments were contingent on students meeting service participation benchmarks and being enrolled for six or more credits (p. 19). The third payment could vary in amount depending on benchmarks attained by the student. Students received $150 for attending an orientation, and $150 for meeting with an Adelante advisor twice during the semester. The remaining $1200 payment was based on academic performance and participation in academic tutoring and workshops. The program orientation was one-hour long at the beginning of the semester. Advising sessions are approximately 20 minutes each. In the first session, students receive advice on their personal education plan. In the second session, students discuss their mid-term progress with their advisor and resolve issues. Adelante students also were incentivized to participate in tutoring, workshops, and platicas sessions (whereby Latino men discuss challenges and issues they face with other Latino men). Students in the intervention group received "thoughtful, positive, specific messages related to academic success, designed to make students feel that they can succeed," (p. 24).
Comparison Group
The comparison group did not receive an Adelante program orientation. They had the opportunity to have general advising offered at the college, but were not incentivized to do so. They could participate in tutoring and academic workshops, but had no scholarship incentive to do so. Comparison group students did not have the option to participate in Platicas sessions. Students in the comparison group receive a broad range of messages to students, but not "thoughtful, positive specific messages related to academic success designed to inspire students to succeed.
Support for implementation
Private foundation support was provided to pay for the research study. Chapter 3 details the implementation and support for the Adelante program. According to the authors, the program components were well-executed. The Adelante program worked closely with staff in Financial Aid, Student Accounts, Student Services, and the Learning Centers to implement the program (p. 35). Students in the program participated at high rates in the advising and student support services offered (p. 35). The college had three full-time staff to oversee the program: a program coordinator, a full-time adviser who worked only with Adelante students, and a support specialist. The college's general advisers also provided in-kind support to Adelante program participants. The college added additional general advisors as the program capacity increased and advisers had up to 35 more students to advise on top of regular, full-time advising responsibilities. The program had oversight from senior administration and strong support, also. Platicas sessions were facilitated by the Program Coordinator (p. 40).
Practice Guide
Review Details
Reviewed: November 2023
- Practice Guide (findings for Student Support Services paired with performance-based scholarship)
- Randomized Controlled Trial
- Meets WWC standards without reservations because it is a randomized controlled trial with low attrition.
This review may not reflect the full body of research evidence for this intervention.
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Findings
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|
|---|---|---|---|---|---|---|---|---|---|
|
Cumulative GPA (3.0-4.0) through 2 semesters |
Student Support Services paired with performance-based scholarship vs. Business as usual |
2 Semesters |
Full sample;
|
30.00 |
31.60 |
No |
-- | ||
| Show Supplemental Findings | |||||||||
|
Grade of C or better in 12+ credits_second semester |
Student Support Services paired with performance-based scholarship vs. Business as usual |
2 Semesters |
Full sample;
|
28.30 |
16.90 |
Yes |
|
||
|
Cumulative GPA (2.0-2.9) through 2 semesters |
Student Support Services paired with performance-based scholarship vs. Business as usual |
2 Semesters |
Full sample;
|
34.90 |
32.10 |
No |
-- | ||
|
second semester term GPA 2.0 or higher |
Student Support Services paired with performance-based scholarship vs. Business as usual |
2 Semesters |
Full sample;
|
34.90 |
32.10 |
No |
-- | ||
|
first semester term GPA 3.0 or higher |
Student Support Services paired with performance-based scholarship vs. Business as usual |
1 Semester |
Full sample;
|
34.80 |
35.40 |
No |
-- | ||
|
second semester term GPA 3.0 or higher |
Student Support Services paired with performance-based scholarship vs. Business as usual |
2 Semesters |
Full sample;
|
26.10 |
28.70 |
No |
-- | ||
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|
|---|---|---|---|---|---|---|---|---|---|
|
cumulative_number of semesters registered |
Student Support Services paired with performance-based scholarship vs. Business as usual |
2 Semesters |
Full sample;
|
1.80 |
1.70 |
Yes |
|
|
|
|
cumulative average number of college-level credits earned |
Student Support Services paired with performance-based scholarship vs. Business as usual |
2 Semesters |
Full sample;
|
13.70 |
12.10 |
Yes |
|
|
|
|
second semester registered for any course |
Student Support Services paired with performance-based scholarship vs. Business as usual |
2 Semesters |
Full sample;
|
78.60 |
74.00 |
No |
-- | ||
| Show Supplemental Findings | |||||||||
|
Second semester average number of college-level credits earned |
Student Support Services paired with performance-based scholarship vs. Business as usual |
2 Semesters |
Full sample;
|
8.30 |
7.30 |
Yes |
|
||
|
First semester average number of college-level credits earned |
Student Support Services paired with performance-based scholarship vs. Business as usual |
1 Semester |
Full sample;
|
6.90 |
6.50 |
No |
-- | ||
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Sample Characteristics
Characteristics of study sample as reported by study author.
-
Male: 100% -
Urban
-
- B
- A
- C
- D
- E
- F
- G
- I
- H
- J
- K
- L
- P
- M
- N
- O
- Q
- R
- S
- V
- U
- T
- W
- X
- Z
- Y
- a
- h
- i
- b
- d
- e
- f
- c
- g
- j
- k
- l
- m
- n
- o
- p
- q
- r
- s
- t
- u
- v
- x
- w
- y
Arizona
-
Race Other or unknown 100% -
Ethnicity Hispanic 100% Not Hispanic or Latino 0% -
Eligible for Free and Reduced Price Lunch Other or unknown 100%
Study Details
Setting
The study took place on six campuses of Pima Community College, a two-year, Hispanic-serving institution with locations throughout the greater Tucson Metropolitan Area in Southern Arizona.
Study sample
All study participants were Latino males with an average age of 24.3 years old. Among them, 36.7% were first-generation college students. About 59% of the students spoke a language other than English at home. About 37.5% of the families received some government benefits, such as food stamps, cash assistance, or welfare.
Intervention Group
The Adelante program was a performance-based scholarship (demonstration) project in which students received scholarships (up to $4500 over three semesters) and other support contingent upon academic progress and achievement in college. There were five main components to the program: 1) An initial program orientation; 2) Advising - each student was assigned to an adviser at the start of his first semester. Advisers used a checklist of topics to cover over their advising sessions and were encouraged to use Spanish appropriately as needed; 3) Tutoring and academic workshops - Students were encouraged to take advantage of tutoring and academic workshops (which they feel were required); 4) Platicas were presentation, discussion, and guest speaker sessions featuring topics and guests of interest to the Adelante students; and 5) the award structure itself in which students were recognized (monetarily) for the level of their participation in events and their academic achievement.
Comparison Group
The comparison condition was business as usual. Students had access to the typical services offered and financial aid available to them at Pima.
Support for implementation
Pima hired three full-time staff members to oversee the day-to-day operations of Adelante: a program coordinator, a full-time adviser, and a support specialist. The program’s advising capacity was augmented by Pima’s general advisers, who volunteered to advise students in Adelante. These advisers had caseloads of up to 35 program group students on top of their regular, full-time advising responsibilities. In addition, the program had oversight from a high-level administrator and support from the college’s leadership.
An indicator of the effect of the intervention, the improvement index can be interpreted as the expected change in percentile rank for an average comparison group student if that student had received the intervention.
For more, please see the WWC Glossary entry for improvement index.
An outcome is the knowledge, skills, and attitudes that are attained as a result of an activity. An outcome measures is an instrument, device, or method that provides data on the outcome.
A finding that is included in the effectiveness rating. Excluded findings may include subgroups and subscales.
The sample on which the analysis was conducted.
The group to which the intervention group is compared, which may include a different intervention, business as usual, or no services.
The timing of the post-intervention outcome measure.
The number of students included in the analysis.
The mean score of students in the intervention group.
The mean score of students in the comparison group.
The WWC considers a finding to be statistically significant if the likelihood that the finding is due to chance alone, rather than a real difference, is less than five percent.
The WWC reviews studies for WWC products, Department of Education grant competitions, and IES performance measures.
The name and version of the document used to guide the review of the study.
The version of the WWC design standards used to guide the review of the study.
The result of the WWC assessment of the study. The rating is based on the strength of evidence of the effectiveness of the intervention. Studies are given a rating of Meets WWC Design Standards without Reservations, Meets WWC Design Standards with Reservations, or >Does Not Meet WWC Design Standards.
A related publication that was reviewed alongside the main study of interest.
Study findings for this report.
Based on the direction, magnitude, and statistical significance of the findings within a domain, the WWC characterizes the findings from a study as one of the following: statistically significant positive effects, substantively important positive effects, indeterminate effects, substantively important negative effects, and statistically significant negative effects. For more, please see the WWC Handbook.
The WWC may review studies for multiple purposes, including different reports and re-reviews using updated standards. Each WWC review of this study is listed in the dropdown. Details on any review may be accessed by making a selection from the drop down list.
Tier 1 Strong indicates strong evidence of effectiveness,
Tier 2 Moderate indicates moderate evidence of effectiveness, and
Tier 3 Promising indicates promising evidence of effectiveness,
as defined in the
non-regulatory guidance for ESSA
and the regulations for ED discretionary grants (EDGAR Part 77).