The statute that authorized the OSP mandated that the Program be evaluated with regard to its impact on student test scores and safety, as well as the "success" of the Program, which we interpret to include satisfaction with school choices. So far, the analysis can only estimate the effects of the Program on these outcomes 1 year after families and students applied to the OSP, or approximately 7 months after the start of students' first school year in the Program.
To estimate the extent to which the Program has an effect on participants, the study first compares the outcomes of the two experimental groups created through random assignment, called the "intent-to-treat" (ITT) approach. The only completely randomized and therefore strictly comparable groups in the study are those students whom the lottery determined were offered scholarships (the treatment group) and those who were not offered scholarships (the control group). The random assignment of students into treatment and control groups should, and did here, produce groups that are similar in key characteristics, both those we can observe and measure (e.g., family income, prior academic achievement) and those we cannot (e.g., motivation to succeed or benefit from the Program). A comparison of these two groups is the most robust and reliable measure of Program impacts because it requires the fewest assumptions to make the groups similar except for their participation in the Program.
The impact analysis proceeded in four steps:
The findings discussed below are robust to adjustments for multiple comparisons and sensitivity tests unless specified.
The analysis suggests the following findings regarding the impacts of a scholarship offer (table ES-2):
Table ES-2. Year 1 Test Score Differential ITT Regression-Based Impact Estimates
The results described above answer the question "what happened to OSP applicants who were offered a scholarship, whether or not a student used the scholarship to attend a private school?" Estimating the impact of using an OSP scholarship involves statistically adjusting the initial impact results to account for two groups of impact sample students: (1) the about 20 percent who received but failed to take up the scholarship offer, who presumably had zero impact from the Program, and (2) an estimated 4 percent in the control group who never received a scholarship offer but who, by virtue of having a sibling with an OSP scholarship, wound up in a participating private school (what we call "program-induced crossover"). These straightforward statistical adjustments yield what are typically called the "impact-on-the-treated" or IOT results. These adjustments increase the size of the scholarship offer effect estimate, but cannot make a statistically insignificant result significant. Therefore, the adjustments are only applied to results that were statistically significant at the scholarship offer stage of the analysis.
The statistically significant findings regarding the use of a scholarship include:
Estimating the effect of attending a private school, regardless of whether an OSP scholarship was used, also begins with the original impact results but uses a more complex statistical procedure6. Because this approach deviates somewhat from the overall experimental design of the evaluation, and yields estimates that are less precise, the private schooling results should be interpreted and used with caution. Like those applied to estimate the impact of OSP scholarship use, the private schooling adjustments increase the size of the scholarship offer effect estimate, but cannot make an insignificant result significant. Therefore, the procedure is only applied to results that were statistically significant at the scholarship offer stage of the analysis.
The main private schooling results suggest that
These results can be placed in the context of other RCTs of scholarship programs for low-income students, which suggest no consistent pattern of academic achievement impacts for the first year of program participation. Among such evaluations of four privately funded scholarship programs, one study of the Charlotte, North Carolina, program clearly found statistically significant overall impacts on math and reading for the first year, while one of three analyses of the New York City program found overall impacts on math achievement (Barnard, Frangakis, Hill, and Rubin, 2003; Greene 2000). When African-Americans are considered separately, a group that makes up nearly 90 percent of the OSP impact study sample, two of three analyses of the New York City program suggest there were achievement gains in math for African-American students in some grade levels (Mayer, Peterson, Myers, Tuttle, and Howell, 2002), but studies of the Dayton, Ohio, and earlier District of Columbia programs found no impacts for this group until students were in the program for 2 years (Howell, Wolf, Campbell, and Peterson, 2002). In contrast, all of the randomized controlled trials that measured parent satisfaction and perceptions of school safety found positive impacts similar to those demonstrated by the OSP the first year (Greene, 2000; Howell and Peterson et al., 2002).
The findings here are based on information collected only a year after students applied to the Program and may not reflect the consistent impacts of the OSP over a longer period of time. Families that apply to voucher programs intend for their children to leave their current public schools and, in the case of the OSP, a much higher share of students in the treatment group (91.3 percent) switched schools—mostly from public to private—compared to those in the control group (56.6 percent). The first-year results, therefore, provide an early look at student experiences in what was a transitional year for most of them. Future reports will examine impacts 2 and 3 years after application to the Program, when any short-term effect of students' transition to new schools may have dissipated. The later reports will also consider additional outcome measures, assess the extent to which school characteristics are associated with impacts, and examine how the DC public school system is changing in response to the Program.
6 The scholarship lottery is used as an instrumental variable (IV) to predict whether a student attended private school. Unlike an indicator variable for actual attendance at a private school, the prediction of private school attendance using the scholarship lottery instrument is unbiased because it is the same for all treatment group students (and all control group students) regardless of their individual enrollment decisions.