The Impact of the Program on Intermediate Outcomes

Understanding the mechanisms through which the OSP does or does not affect student outcomes requires examining the expectations, experiences, and educational environments made possible by Program participation. The analysis here estimates the impact of the Program on a set of “intermediate outcomes” that are influenced by parents’ choice of whether to use an OSP scholarship and where to use it, but are not end outcomes themselves. The method used to estimate the impacts on intermediate outcomes is identical to that used to estimate impacts on the key Program outcomes, such as academic achievement.

Prior to data analysis, possible intermediate outcomes of the OSP were selected based on existing research and theory regarding scholarship programs and educational achievement. Because 24 intermediate outcome candidates were identified through this process, the variables were organized into four conceptual groups or clusters to aid in the analysis.⁸

There is no way to rigorously evaluate the linkages between the intermediate outcomes and achievement—students are not randomly assigned to the experience of various educational conditions and programs. That is why any findings from this element of the study do not suggest that we have learned what specific factors “caused” any observed test score impacts, only that certain factors emerge from the analysis as possible candidates for mediating influence. The analyses are exploratory, and, given the number of factors analyzed, some of the statistically significant findings may be “false discoveries” (due to chance).

Overall, 2 years after applying for a scholarship, the Program had an impact on 10 of the 24 intermediate outcomes, 8 of which remained statistically significant after adjustments for multiple comparisons:

• Home Educational Supports. The results suggest that the Program may have had an impact on two of four intermediate outcomes in this group. The Program appeared to produce a positive impact on parents’ aspirations for how far in school their child would go (ES = .12); however, this result may be a false discovery. The Program led to students’ experiencing more time spent commuting to school from their homes (ES = .25), a result that did not lose statistical significance after adjustments for multiple comparisons. There were no statistically significant differences between the treatment and control groups on the involvement in school reported by parents in year 2 (ES = -06) or on the use of a tutor outside of school (ES = -07).
• Student Motivation and Engagement. The Program had no statistically significant impacts on any of the six elements of this group of intermediate outcomes. Two years after they applied to the OSP, the treatment and control group students reported similar aspirations for future schooling (ES = -.11), frequency of doing homework (ES = -.10), time spent reading for fun (ES = .02), and engagement in extracurricular activities (ES = .08). There were no statistically significant differences in student attendance (ES = -.11) or tardiness rates (ES = -.11), as reported by parents.
• Instructional Characteristics. The offer of a scholarship appears to have had a statistically significant impact on 5 of the 10 intermediate outcomes in this group. Being offered a scholarship led to students’ experiencing smaller classes, as measured by student/teacher ratios (ES = -.29). The Program also led to students’ experiencing a lower likelihood that their school offered either tutoring (ES = -.32) or special programs for children who were English language learners or had learning problems (ES = -.66). At the same time, however, the Program had a positive impact on the use of an in-school tutor, presumably in schools that made them available (ES = .13). The OSP also led to students’ experiencing a higher likelihood of being in a school that offered enrichment programs (ES = .19). The statistical significance of these five results was not affected by adjustments for multiple comparisons. There were no differences between the treatment and control groups in how students rated their teacher’s attitude (ES = .02) or the challenge of their classes (ES = -.04), the school’s use of ability grouping (ES = .13), the availability of programs for advanced learners (ES = .12), or before- and after-school programs (ES = .04).
• School Environment. The Program may have affected three of the four measures of school environment. Students in the treatment group experienced schools that were smaller (ES = -.43) and had a smaller percentage of non-white students (ES = -.39) than the schools of the control group, findings that were not affected by adjustments for multiple comparisons. Treatment group students also reported having better behaved peers in the classroom than did control group students (ES = .16), although adjustments for multiple comparisons suggest that this finding may be a false discovery. There were no differences in parents’ reports of how their child’s school communicates with them (ES = .01).

It is important to note that the findings regarding the impacts of the OSP reflect the particular Program elements that evolved from the law passed by Congress, and the characteristics of students, families, and schools—public and private—that exist in the Nation’s capital. The same program implemented in another city could yield different results, and a different scholarship program in Washington, DC, might also produce different outcomes.

Top

---

⁸ Intermediate Outcome Conceptual Grouping 1, Home Educational Supports, includes parent involvement, parent aspirations, out-of-school tutor usage, and school transit time. Intermediate Outcome Conceptual Grouping 2, Student Motivation and Engagement, includes student aspirations, attendance, tardiness, reading for fun, engagement in extracurricular activities, and frequency of homework. Intermediate Outcome Conceptual Grouping 3, Instructional Characteristics, includes student/teacher ratio, teacher attitude, challenge of classes, ability grouping, availability of tutors, in-school tutor usage, programs to assist students with learning disabilities or English language learners, programs for advanced learners, before-/after-school care programs, and enrichment programs. Intermediate Outcome Conceptual Grouping 4, School Environment, includes parent/ school communication, school size, percent non-white, and peer classroom behavior.