Skip Navigation
Evaluation of the DC Opportunity Scholarship Program:

NCEE 2009-4050
March 2009

The Impact of the Program on Intermediate Outcomes

Understanding the mechanisms through which the OSP does or does not affect student outcomes requires examining the expectations, experiences, and educational environments made possible by Program participation. The analysis here estimates the impact of the Program on a set of "intermediate outcomes" that may be influenced by parents' choice of whether to use an OSP scholarship and where to use it, but are not end outcomes themselves. The method used to estimate the impacts on intermediate outcomes is identical to that used to estimate impacts on the key Program outcomes, such as academic achievement.

Prior to data analysis, possible intermediate outcomes of the OSP were selected based on existing research and theory regarding scholarship programs and educational achievement. Because 24 intermediate outcome candidates were identified through this process, the variables were organized into four conceptual groups or clusters, as described below, to aid in the analysis.

There is no way to rigorously evaluate the linkages between the intermediate outcomes and achievement–students are not randomly assigned to the experience of various educational conditions and programs. That is why any findings from this element of the study do not suggest that we have learned what specific factors "caused" any observed test score impacts, only that certain factors emerge from the analysis as possible candidates for mediating influence because the Program affected students' experience of these factors. The analyses are exploratory, and, given the number of factors analyzed, some of the statistically significant findings may be "false discoveries" (due to chance).

Overall, 3 years after applying to the Program, the offer of an Opportunity Scholarship appears to have had an impact on 8 of the 24 intermediate outcomes examined, 7 of which remained statistically significant after adjustments for multiple comparisons:

  • Home Educational Supports. Of the four intermediate outcomes in this category, the offer of a scholarship had an impact on one of them. There was a significant negative impact on tutor usage outside of school (ES = -.14), and this impact remained statistically significant after adjustments for multiple comparisons. There were no statistically significant differences between the treatment and control groups on parents' reports of their involvement in school in year 3 (ES = -.11), parents' aspirations for how far in school their children would go (ES = .02), or time required for the student to get to school (odds ratio = 1.13).11
  • Student Motivation and Engagement. Of the six intermediate outcomes in this category, the offer of a scholarship may have had an impact on one of them. Based on student surveys, the offer of a scholarship seems to have had a significant negative impact on whether students read for fun (ES = -.16). Adjustments for multiple comparisons, however, indicate that this result could be a false discovery, so it should be interpreted with caution. There were no statistically significant differences between the treatment and control groups in their reported aspirations for future schooling (ES = -.14), engagement in extracurricular activities (ES = .04), and frequency of doing homework (ES = .08), or in their parents' reports of student attendance (odds ratio = 1.11) or tardiness rates (odds ratio = 1.19).
  • Instructional Characteristics. The offer of a scholarship had a statistically significant impact on 5 of the 10 intermediate outcomes in this group of indicators. Students offered a scholarship experienced a lower likelihood that their school offered tutoring (ES = -.38), special programs for children who were English language learners (ES = -.61), or special programs for students with learning problems (ES = -.36) compared to control group students; these impacts remained statistically significant after adjustments for multiple comparisons. Students offered a scholarship experienced a higher likelihood that their school offered programs for advanced learners (ES = .27) and such enrichment programs as art, music, and foreign language (ES = .23); these two impact estimates also remained statistically significant after adjustments for multiple comparisons. There were no significant differences between the treatment and control groups in student/teacher ratio (ES = .01), how students rated their teacher's attitude (ES = -.04), the school's use of ability grouping (ES = .02), in-school tutor usage (ES = .04), or the availability of before- and after-school programs (ES = -.11).
  • School Environment. The offer of a scholarship affected one of four measures of school environment. Students offered a scholarship experienced schools that were smaller by an average of 182 students (ES = -.29) than the schools attended by students in the control group; this impact remained statistically significant after adjustments for multiple comparisons. There were no statistically significant differences between the treatment and control groups, on average, in school reports of parent/school communication practices (ES = -.06), the percentage of minority students at the school (ES = -.10), or the classroom behavior of peers (ES = .09) based on student reports.

It is important to note that the findings regarding the impacts of the OSP reflect the particular Program elements that evolved from the law passed by Congress and the characteristics of students, families, and schools–public and private–that exist in the Nation's capital. The same program implemented in another city could yield different results, and a scholarship program in Washington, DC, with different design features than the OSP might also produce different outcomes.

Top

11 The effect size for this categorical variable is expressed as an odds ratio, which describes the extent to which being in the treatment group increases (if above 1.0) or decreases (if below 1.0) the likelihood of giving a higher-category response.