The Impact of the Program on Intermediate Outcomes
Understanding the mechanisms through which the OSP does or does not affect student
outcomes requires examining the expectations, experiences, and educational environments
made possible by Program participation. The analysis here estimates the impact of
the Program on a set of "intermediate outcomes" that may be influenced by parents'
choice of whether to use an OSP scholarship and where to use it, but are not end
outcomes themselves. The method used to estimate the impacts on intermediate outcomes
is identical to that used to estimate impacts on the key Program outcomes, such
as academic achievement.
Prior to data analysis, possible intermediate outcomes of the OSP were selected
based on existing research and theory regarding scholarship programs and educational
achievement. Because 24 intermediate outcome candidates were identified through
this process, the variables were organized into four conceptual groups or clusters,
as described below, to aid in the analysis.
There is no way to rigorously evaluate the linkages between the intermediate outcomes
and achievement–students are not randomly assigned to the experience of various
educational conditions and programs. That is why any findings from this element
of the study do not suggest that we have learned what specific factors "caused"
any observed test score impacts, only that certain factors emerge from the analysis
as possible candidates for mediating influence because the Program affected students'
experience of these factors. The analyses are exploratory, and, given the number
of factors analyzed, some of the statistically significant findings may be "false
discoveries" (due to chance).
Overall, 3 years after applying to the Program, the offer of an Opportunity Scholarship
appears to have had an impact on 8 of the 24 intermediate outcomes examined, 7 of
which remained statistically significant after adjustments for multiple comparisons:
- Home Educational Supports. Of the four intermediate outcomes
in this category, the offer of a scholarship had an impact on one of them. There
was a significant negative impact on tutor usage outside of school (ES = -.14),
and this impact remained statistically significant after adjustments for multiple
comparisons. There were no statistically significant differences between the treatment
and control groups on parents' reports of their involvement in school in year 3
(ES = -.11), parents' aspirations for how far in school their children would go
(ES = .02), or time required for the student to get to school (odds ratio = 1.13).11
- Student Motivation and Engagement. Of the six intermediate
outcomes in this category, the offer of a scholarship may have had an impact on
one of them. Based on student surveys, the offer of a scholarship seems to have
had a significant negative impact on whether students read for fun (ES = -.16).
Adjustments for multiple comparisons, however, indicate that this result could be
a false discovery, so it should be interpreted with caution. There were no statistically
significant differences between the treatment and control groups in their reported
aspirations for future schooling (ES = -.14), engagement in extracurricular activities
(ES = .04), and frequency of doing homework (ES = .08), or in their parents' reports
of student attendance (odds ratio = 1.11) or tardiness rates (odds ratio = 1.19).
- Instructional Characteristics. The offer of a scholarship
had a statistically significant impact on 5 of the 10 intermediate outcomes in this
group of indicators. Students offered a scholarship experienced a lower likelihood
that their school offered tutoring (ES = -.38), special programs for children who
were English language learners (ES = -.61), or special programs for students with
learning problems (ES = -.36) compared to control group students; these impacts
remained statistically significant after adjustments for multiple comparisons. Students
offered a scholarship experienced a higher likelihood that their school offered
programs for advanced learners (ES = .27) and such enrichment programs as art, music,
and foreign language (ES = .23); these two impact estimates also remained statistically
significant after adjustments for multiple comparisons. There were no significant
differences between the treatment and control groups in student/teacher ratio (ES
= .01), how students rated their teacher's attitude (ES = -.04), the school's use
of ability grouping (ES = .02), in-school tutor usage (ES = .04), or the availability
of before- and after-school programs (ES = -.11).
- School Environment. The offer of a scholarship affected
one of four measures of school environment. Students offered a scholarship experienced
schools that were smaller by an average of 182 students (ES = -.29) than the schools
attended by students in the control group; this impact remained statistically significant
after adjustments for multiple comparisons. There were no statistically significant
differences between the treatment and control groups, on average, in school reports
of parent/school communication practices (ES = -.06), the percentage of minority
students at the school (ES = -.10), or the classroom behavior of peers (ES = .09)
based on student reports.
It is important to note that the findings regarding the impacts of the OSP reflect
the particular Program elements that evolved from the law passed by Congress and
the characteristics of students, families, and schools–public and private–that
exist in the Nation's capital. The same program implemented in another city could
yield different results, and a scholarship program in Washington, DC, with different
design features than the OSP might also produce different outcomes.