Impact of the Program After 3 Years: Key Outcomes
The statute that authorized the OSP mandated that the Program be evaluated with
regard to its impact on student test scores and school safety, as well as the "success"
of the Program, which, in the design of this study, includes satisfaction with school
choices. The impacts of the Program on these outcomes are presented in two ways:
(1) the impact of the offer of an OSP scholarship, derived straight from comparing
outcomes of the treatment and control groups, and (2) the impact of using an OSP
scholarship, calculated from the unbiased treatment-control group comparison, but
statistically adjusting for students who declined to use their scholarships.6 The main focus of this study was on the overall
group of students, with a secondary interest in students who applied from SINI schools,
followed by other subgroups of students (e.g., defined by their academic performance
at application, their gender, or their grade level).
Previous reports released in spring 2007 and spring 2008 indicated that 1 and 2
years after application, there were no statistically significant impacts on overall
academic achievement or on student perceptions of school safety or satisfaction
(Wolf et al. 2007; Wolf et al. 2008). Parents were more satisfied if their child
was in the Program and viewed their child's school as safer and more orderly. Among
the secondary analyses of subgroups, there were impacts on math test scores in year
1 for students who applied from non-SINI schools and those with relatively higher
pre-Program test scores, and impacts in reading test scores (but not math) in year
2 for those same two subgroups plus students who applied in the first year of Program
implementation. However, these findings were no longer statistically significant
when subjected to a reliability test to adjust for the multiple comparisons of treatment
and control group students across 10 subgroups; the results may be "false discoveries"
and should therefore be interpreted and used with caution. Throughout this report,
the phrases "appears to have an impact" and "may have had an impact" are used to
caution readers regarding statistically significant impacts that may have been false
The analyses in this report were conducted using data collected on students 3 years
after they applied to the OSP.7
Impacts on Students and Parents Overall
- Across the full sample, there was a statistically significant impact on reading
achievement of 4.5 scale score points (effect size (ES) = .13)8 from the offer of a scholarship and 5.3 scale score points
(ES = .15) from the use of a scholarship (table 3). These impacts are equivalent
to 3.1 and 3.7 months of additional learning, respectively.9
- There was no statistically significant impact on math achievement, overall (ES =
.03) from the offer of a scholarship nor from the use of a scholarship (table 3).10
- Parents of students offered a scholarship were more likely to report their child's
school to be safer and have a more orderly school climate (ES = .29) compared to
parents of students not offered a scholarship (figure 3); the same was true for
parents of students who chose to use their scholarships (ES = .34).
- On the other hand, students who were offered a scholarship reported similar levels
of school safety and an orderly climate compared to those in the control group (ES
= .06; figure 3); there was also no significant impact on student reports of school
safety and an orderly climate from using a scholarship (ES = .07).
- The Program produced a positive impact on parent satisfaction with their child's
school as measured by the likelihood of grading the school an "A" or "B," both for
the impact of a scholarship offer (ES = .22; figure 4) and the impact of scholarship
use (ES = .26).
Impacts on Subgroups
In addition to determining the general impacts of the OSP on all study participants,
this evaluation also reports programmatic impacts on policy-relevant subgroups of
students. The subgroups were designated prior to data collection and include students
who were attending SINI versus non-SINI schools at application, those relatively
higher or lower performing at baseline, girls or boys, elementary versus high school
students, and those from application cohort 1 or cohort 2. Since the subgroup analysis
involves significance tests across multiple comparisons of treatment and control
students, some of which may be statistically significant merely by chance, these
subgroup-specific results should be interpreted with caution. Specifically:
Subgroup Achievement Impacts
- There were no statistically significant reading (ES = .05) or math (ES = .01) achievement
impacts for the high-priority subgroup of students who had attended a SINI public
school under No Child Left Behind (NCLB) before applying to the Program.
- There were statistically significant impacts on reading test scores in year 3 for
five subgroups of students, although the statistical significance of two of the
subgroup findings was not robust to adjustments for multiple comparisons:
- Students who attended non-SINI public schools prior to application to the Program
(56 percent of the impact sample) scored an average of 6.6 scale score points higher
in reading (ES = .19) if they were offered the scholarship compared to not being
offered a scholarship and 7.7 scale score points higher (ES = .22) if they used
their scholarship compared to not being offered a scholarship. These scale score
differences between the treatment and control groups translate into 4.1 and 4.9
additional months of learning, or half a year of schooling based on a typical 9-month
- Students who entered the Program in the higher two-thirds of the test-score performance
distribution at baseline (66 percent of the impact sample) scored an average of
5.5 scale score points higher in reading (ES = .17) if they were offered a scholarship
and 6.2 scale score points higher (ES = .19) if they used their scholarship, impacts
equivalent to 4.0 and 4.6 months of learning gains.
- Female students scored an average of 5.1 scale score points higher in reading (ES
= .15) if they were offered a scholarship and 5.8 scale score points higher (ES
= .17) if they used their scholarship. These impacts represent 3.1 and 3.6 months
of additional learning, respectively. The statistical significance of this finding
was not robust to adjustments for multiple comparisons.
- Students who entered the Program in grades K-8 (81 percent of the impact sample)
scored an average of 5.2 scale score points higher in reading (ES = .15) or 2.9
months of additional learning if they were offered a scholarship compared to not
being offered a scholarship and 6.0 scale score points higher (ES = .17) or 3.3
months of additional learning if they used their scholarship compared to not being
offered a scholarship.
- Students from the first cohort of applicants (21 percent of the impact sample) scored
an average of 8.7 scale score points higher in reading (ES = .31) if they were offered
a scholarship compared to not being offered a scholarship and 11.7 scale score points
higher (ES = .42) if they used their scholarship compared to not being offered a
scholarship. These impacts translate into 14.1 and 18.9 months of additional learning
(1.5 to 2 years of typical schooling). The statistical significance of this finding
was not robust to adjustments for multiple comparisons.
- The OSP had no statistically significant reading impacts for other subgroups of
participating students, including those in the lower third of the test-score performance
distribution at baseline, boys, secondary students, and students from the second
cohort of applicants (ES ranging from -.00 to .11).
- The OSP had no statistically significant math impacts for any of the 10 subgroups
(ES ranging from -.16 to .23).
Subgroup Safety and Satisfaction Impacts
- All of the 10 subgroups analyzed, including parents of the high-priority subgroup
of students who had attended SINI schools at baseline, reported viewing their child's
school as safer and more orderly if the child was offered or using an OSP scholarship
compared to not being offered a scholarship. Effect sizes for the impact of an offer
of a scholarship on parent perceptions of safety and an orderly school climate for
the 10 subgroups ranged from .27 to .40. Adjustments for multiple comparisons indicate
that these 10 subgroup impacts on parental perceptions of safety and school climate
are not likely to be false discoveries.
- Consistent with the finding for students overall, none of the subgroups of students
reported experiencing differences in safety and an orderly school climate if they
were offered (ES range from -.03 to .08) or using an OSP scholarship.
- In addition to an overall impact on parental satisfaction with their child's school,
the Program produced satisfaction impacts on 7 of the 10 subgroups analyzed. Effect
sizes for the impact of an offer of a scholarship on the likelihood of a parent
grading his/her child's school "A" or "B" for these seven subgroups ranged from
.16 to .41. Adjustments for multiple comparisons indicated that none of these parent
satisfaction subgroup impacts may have been a false discovery. The parents of students
who had attended SINI schools, parents of students in the lower one-third of the
test score distribution, and parents of high school students generally did not report
higher levels of school satisfaction that were statistically significant as a result
of the treatment (ES ranged from -.03 to .13).
- There were no statistically significant differences between the treatment group
and the control group for all 10 subgroups in the likelihood that students gave
their school a grade of A or B (ES ranged from -.18 to .05).