Impact of the Program After 2 Years: Key Outcomes

The statute that authorized the OSP mandated that the Program be evaluated with regard to its impact on student test scores and school safety, as well as the “success” of the Program, which, in the design of this study, includes satisfaction with school choices. The impacts of the Program on these outcomes are presented in two ways: (1) the impact of the offer of an OSP scholarship, derived straight from comparing outcomes of the treatment and control groups, and (2) the impact of using an OSP scholarship, calculated from the unbiased treatment-control group comparison, but statistically netting out students who declined to use their scholarships.⁵ The main focus of this study was on the overall group of students, with a secondary interest in students who applied from SINI schools, followed by other subgroups of students (e.g., defined by their academic performance at application, their gender, or their grade level).

A previous report released in spring 2007 indicated that 1 year after application there were no statistically significant impacts on overall academic achievement or on student perceptions of school safety or satisfaction (Wolf et al. 2007). Parents were more satisfied if their child was in the Program and viewed their child’s school as less dangerous. Among the secondary analyses of subgroups, there were impacts on math for students who applied from non-SINI schools and for those with relatively higher pre-Program test scores. Statistical adjustments for multiple comparisons suggested there is a possibility that the subgroup achievement impacts in year 1 were chance discoveries.

The analyses in this report were conducted using data collected on students 2 years after they applied to the OSP.

Impacts on Students and Parents Overall

• Across the full sample, there were no statistically significant impacts on reading achievement (effect size (ES) = .09)⁶ or math achievement (ES = .01) from the offer of a scholarship (table 3) nor from the use of a scholarship.⁷
• Parents of students offered a scholarship were less likely to report serious concerns about school danger (ES = -.27) compared to parents of students not offered a scholarship (table 4); the same was true for parents of students who chose to use their scholarships (ES = -.34).
• On the other hand, students who were offered a scholarship reported similar levels of dangerous activities at school compared to those in the control group (ES = -.01; table 4); there was also no impact on student reports of school safety from using a scholarship (ES = -.01).
• The Program produced a positive impact on parent satisfaction with their child’s school, for example regarding the likelihood of grading the school an “A” or “B,” both for the impact of a scholarship offer (ES = .26; table 5) and the impact of scholarship use (ES = .33).
• Overall, there were no impacts of the OSP from being offered (ES = .05 to .13; table 5) or using a scholarship on students’ satisfaction with his or her school.

Impacts on Subgroups

In addition to determining the general impacts of the OSP on all study participants, this evaluation also reports Programmatic impacts on policy-relevant subgroups of students. The subgroups were designated prior to data collection and include students who were attending SINI versus non-SINI schools at application, those relatively higher or lower performing at baseline, girls or boys, elementary versus high school students, and those from application cohort 1 or cohort 2. Since the subgroup analysis involves significance tests across multiple comparisons of treatment and control students, some of which may be statistically significant merely by chance, these subgroup-specific results should be interpreted with caution. Specifically:

Subgroup Achievement Impacts

• There were no statistically significant reading (ES = -.00) or math (ES = .05) achievement impacts for the high-priority subgroup of students who had attended a SINI public school under No Child Left Behind (NCLB) before applying to the Program.
• The Program may have had a positive impact on reading test scores in year 2 for three subgroups of students, although the statistical significance of the findings was not robust to adjustments for multiple comparisons:
  • Students who attended non-SINI public schools prior to application to the Program (56 percent of the impact sample) scored an average of 5.7 scale score points higher in reading (ES = .15) if they were offered the scholarship compared to not being offered a scholarship and 6.9 scale score points higher (ES = .18) if they used their scholarship compared to not being offered a scholarship.
  • Students who entered the Program in the higher two-thirds of the test-score performance distribution at baseline (66 percent of the impact sample) scored an average of 5.2 scale score points higher in reading (ES = .15) if they were offered a scholarship compared to not being offered a scholarship and 6.3 scale score points higher (ES = .18) if they used their scholarship compared to not being offered a scholarship.
  • Students from the first cohort of applicants (21 percent of the impact sample) scored an average of 8.7 scale score points higher in reading (ES = .27) if they were offered a scholarship compared to not being offered a scholarship and 12.2 scale score points higher (ES = .37) if they used their scholarship compared to not being offered a scholarship.
• The OSP had no statistically significant achievement impacts for other subgroups of participating students, including those in the lower third of the test-score performance distribution at baseline, boys, girls, elementary students, secondary students, and students from the second cohort of applicants (effect sizes ranging from -.14 to .11).

Subgroup Safety and Satisfaction Impacts

• Eight of the 10 subgroups analyzed, including parents of the high-priority subgroup of students who had attended SINI schools, reported viewing their child’s school as less dangerous if the child was offered or using an OSP scholarship compared to not being offered a scholarship. Effect sizes for the impact of an offer of a scholarship on parent perceptions of school danger for the eight affected subgroups ranged from -.21 to -.35. Adjustments for multiple comparisons indicate that these eight subgroup impacts on parental perceptions of safety are not likely to be false discoveries. The parents of students who were relatively lower performing at baseline and those in high school were the exceptions, as they did not report lower or different levels of perceived school danger as a result of the treatment.
• Consistent with the finding for students overall, none of the subgroups of students reported experiencing differences in dangerous activities at school if they were in the Program. Thus, there was no impact on students’ perceptions of school safety from either the offer or the use of a scholarship for any of the subgroups (effect sizes range from -.11 to .09).
• In addition to an overall impact on parental satisfaction with their child’s school, the Program produced satisfaction impacts on 8 of the 10 subgroups analyzed, including the high-priority subgroup of parents of students who had attended SINI schools. Effect sizes for the impact of an offer of a scholarship on the likelihood of a parent grading their child’s school “A” or “B” for the eight affected subgroups ranged from .18 to .34. Adjustments for multiple comparisons indicate that one of these eight subgroup impacts (for the parents of students who were relatively lower performing at baseline) may have been a false discovery. The statistical significance of the other seven subgroup impacts on parent satisfaction with their child’s school was not affected by adjustments for multiple comparisons. The parents of high school students and those in the first cohort of applicants generally did not report higher levels of school satisfaction that were statistically significant as a result of the treatment (effect sizes range from .02 to .18).
• With one exception, there was no impact on school satisfaction if students were offered a scholarship, across subgroups. The high-priority subgroup of students who applied from a SINI school were more likely to give their school a grade of A or B (ES = .24) if they were offered a scholarship compared to not being offered a scholarship, although adjustments for multiple comparisons indicate that this finding may be a false discovery.

Top

---

⁵ This analysis uses straightforward statistical adjustments to account not only for the approximately 25 percent of impact sample respondents who received the offer of a scholarship but declined to use it (the "decliners"), but also the estimated 2.3 percent of the control group who never received a scholarship offer but who, by virtue of having a sibling with an OSP scholarship, ended up in a participating private school (we call this "program-enabled crossover"). These adjustments increase the size of the scholarship offer effect estimates, but cannot make a statistically insignificant result significant.
⁶ An effect size (ES) is a standardized measure of the relative size of a program impact. In this report, effect sizes are expressed as a proportion of a standard deviation of the distribution of values observed for the study control group. One full standard deviation above and below the average value for a variable such as outcome test scores contains 64 percent of the observations in the distribution. Two full standard deviations above and below the average contain 95 percent of the observations.
⁷ The magnitudes of these estimated achievement effects are below the threshold of .11 standard deviations, estimated by the power analysis to be the study’s Minimum Detectable Effect size.