Impact of the Program After 2 Years: Key Outcomes
The statute that authorized the OSP mandated that the Program be evaluated with regard to
its impact on student test scores and school safety, as well as the “success” of the Program, which, in the
design of this study, includes satisfaction with school choices. The impacts of the Program on these
outcomes are presented in two ways: (1) the impact of the offer of an OSP scholarship, derived straight
from comparing outcomes of the treatment and control groups, and (2) the impact of using an OSP
scholarship, calculated from the unbiased treatment-control group comparison, but statistically netting out
students who declined to use their scholarships.5 The main focus of this study was on the overall group of students, with a secondary interest in students who applied from SINI schools, followed by other
subgroups of students (e.g., defined by their academic performance at application, their gender, or their
grade level).
A previous report released in spring 2007 indicated that 1 year after application there were
no statistically significant impacts on overall academic achievement or on student perceptions of school safety or satisfaction (Wolf et al. 2007). Parents were more satisfied if their child was in the Program and viewed their child’s school as less dangerous. Among the secondary analyses of subgroups, there were impacts on math for students who applied from non-SINI schools and for those with relatively higher pre-Program test scores. Statistical adjustments for multiple comparisons suggested there is a possibility that the subgroup achievement impacts in year 1 were chance discoveries.
The analyses in this report were conducted using data collected on students 2 years after they
applied to the OSP.
Impacts on Students and Parents Overall
- Across the full sample, there were no statistically significant impacts on reading
achievement (effect size (ES) = .09)6 or math achievement (ES = .01) from the offer of a scholarship (table 3) nor from the use of a scholarship.7
- Parents of students offered a scholarship were less likely to report serious concerns
about school danger (ES = -.27) compared to parents of students not offered a
scholarship (table 4); the same was true for parents of students who chose to use their
scholarships (ES = -.34).
- On the other hand, students who were offered a scholarship reported similar levels of
dangerous activities at school compared to those in the control group (ES = -.01;
table 4); there was also no impact on student reports of school safety from using a scholarship (ES = -.01).
- The Program produced a positive impact on parent satisfaction with their child’s
school, for example regarding the likelihood of grading the school an “A” or “B,” both for the impact of a scholarship offer (ES = .26; table 5) and the impact of scholarship use (ES = .33).
- Overall, there were no impacts of the OSP from being offered (ES = .05 to .13; table 5) or using a scholarship on students’ satisfaction with his or her school.
Impacts on Subgroups
In addition to determining the general impacts of the OSP on all study participants, this
evaluation also reports Programmatic impacts on policy-relevant subgroups of students. The subgroups
were designated prior to data collection and include students who were attending SINI versus non-SINI
schools at application, those relatively higher or lower performing at baseline, girls or boys, elementary
versus high school students, and those from application cohort 1 or cohort 2. Since the subgroup analysis
involves significance tests across multiple comparisons of treatment and control students, some of which
may be statistically significant merely by chance, these subgroup-specific results should be interpreted
with caution. Specifically:
Subgroup Achievement Impacts
- There were no statistically significant reading (ES = -.00) or math (ES = .05)
achievement impacts for the high-priority subgroup of students who had attended a
SINI public school under No Child Left Behind (NCLB) before applying to the
Program.
- The Program may have had a positive impact on reading test scores in year 2 for three
subgroups of students, although the statistical significance of the findings was not
robust to adjustments for multiple comparisons:
- Students who attended non-SINI public schools prior to application to the
Program (56 percent of the impact sample) scored an average of 5.7 scale score
points higher in reading (ES = .15) if they were offered the scholarship
compared to not being offered a scholarship and 6.9 scale score points higher
(ES = .18) if they used their scholarship compared to not being offered a
scholarship.
- Students who entered the Program in the higher two-thirds of the test-score
performance distribution at baseline (66 percent of the impact sample) scored an
average of 5.2 scale score points higher in reading (ES = .15) if they were
offered a scholarship compared to not being offered a scholarship and 6.3 scale
score points higher (ES = .18) if they used their scholarship compared to not
being offered a scholarship.
- Students from the first cohort of applicants (21 percent of the impact sample)
scored an average of 8.7 scale score points higher in reading (ES = .27) if they
were offered a scholarship compared to not being offered a scholarship and 12.2
scale score points higher (ES = .37) if they used their scholarship compared to
not being offered a scholarship.
- The OSP had no statistically significant achievement impacts for other subgroups of
participating students, including those in the lower third of the test-score performance
distribution at baseline, boys, girls, elementary students, secondary students, and
students from the second cohort of applicants (effect sizes ranging from -.14 to .11).
Subgroup Safety and Satisfaction Impacts
- Eight of the 10 subgroups analyzed, including parents of the high-priority subgroup of
students who had attended SINI schools, reported viewing their child’s school as less
dangerous if the child was offered or using an OSP scholarship compared to not being
offered a scholarship. Effect sizes for the impact of an offer of a scholarship on parent
perceptions of school danger for the eight affected subgroups ranged from -.21 to -.35.
Adjustments for multiple comparisons indicate that these eight subgroup impacts on
parental perceptions of safety are not likely to be false discoveries. The parents of
students who were relatively lower performing at baseline and those in high school
were the exceptions, as they did not report lower or different levels of perceived school
danger as a result of the treatment.
- Consistent with the finding for students overall, none of the subgroups of students
reported experiencing differences in dangerous activities at school if they were in the
Program. Thus, there was no impact on students’ perceptions of school safety from
either the offer or the use of a scholarship for any of the subgroups (effect sizes range
from -.11 to .09).
- In addition to an overall impact on parental satisfaction with their child’s school, the
Program produced satisfaction impacts on 8 of the 10 subgroups analyzed, including
the high-priority subgroup of parents of students who had attended SINI schools.
Effect sizes for the impact of an offer of a scholarship on the likelihood of a parent
grading their child’s school “A” or “B” for the eight affected subgroups ranged from
.18 to .34. Adjustments for multiple comparisons indicate that one of these eight
subgroup impacts (for the parents of students who were relatively lower performing at
baseline) may have been a false discovery. The statistical significance of the other
seven subgroup impacts on parent satisfaction with their child’s school was not
affected by adjustments for multiple comparisons. The parents of high school students
and those in the first cohort of applicants generally did not report higher levels of
school satisfaction that were statistically significant as a result of the treatment (effect
sizes range from .02 to .18).
- With one exception, there was no impact on school satisfaction if students were offered
a scholarship, across subgroups. The high-priority subgroup of students who applied
from a SINI school were more likely to give their school a grade of A or B (ES = .24)
if they were offered a scholarship compared to not being offered a scholarship,
although adjustments for multiple comparisons indicate that this finding may be a false
discovery.
Top
5 This analysis uses straightforward statistical adjustments to account not only for the approximately 25 percent of impact sample respondents who received the offer of a scholarship but declined to use it (the "decliners"), but also the estimated 2.3 percent of the control group who never received a scholarship offer but who, by virtue of having a sibling with an OSP scholarship, ended up in a participating private school (we call this "program-enabled crossover"). These adjustments increase the size of the scholarship offer effect estimates, but cannot make a statistically insignificant result significant.
6 An effect size (ES) is a standardized measure of the relative size of a program impact. In this report, effect sizes are expressed as a proportion of a standard deviation of the distribution of values observed for the study control group. One full standard deviation above and below the average value for a variable such as outcome test scores contains 64 percent of the observations in the distribution. Two full standard deviations above and below the average contain 95 percent of the observations.
7 The magnitudes of these estimated achievement effects are below the threshold of .11 standard deviations, estimated by the power analysis to be the study’s Minimum Detectable Effect size.