Skip Navigation
Evaluation of the DC Opportunity Scholarship Program: Impacts After One Year
NCEE 2007-4009
June 2007

The Impact of the Program After 1 Year

The statute that authorized the OSP mandated that the Program be evaluated with regard to its impact on student test scores and safety, as well as the "success" of the Program, which we interpret to include satisfaction with school choices. So far, the analysis can only estimate the effects of the Program on these outcomes 1 year after families and students applied to the OSP, or approximately 7 months after the start of students' first school year in the Program.

Impact of Being Awarded a Scholarship (Experimental Estimates)

To estimate the extent to which the Program has an effect on participants, the study first compares the outcomes of the two experimental groups created through random assignment, called the "intent-to-treat" (ITT) approach. The only completely randomized and therefore strictly comparable groups in the study are those students whom the lottery determined were offered scholarships (the treatment group) and those who were not offered scholarships (the control group). The random assignment of students into treatment and control groups should, and did here, produce groups that are similar in key characteristics, both those we can observe and measure (e.g., family income, prior academic achievement) and those we cannot (e.g., motivation to succeed or benefit from the Program). A comparison of these two groups is the most robust and reliable measure of Program impacts because it requires the fewest assumptions to make the groups similar except for their participation in the Program.

The impact analysis proceeded in four steps:

  1. The impacts of the program on each outcome of interest were estimated for the entire sample of study participants, using an analytic model and well-established statistical approaches that were specified in advance.
  2. Those same impacts were estimated for various policy-relevant subgroups of participants that differed based on the "need of improvement" status of their school (SINI), their baseline academic performance, their gender, their schooling level, and their cohort status.
  3. A reliability test was administered to the results drawn from multiple comparisons of treatment and control group members (e.g., across 10 different subgroups) to identify any statistically significant findings that could be due to chance, or what statisticians refer to as "false discoveries."
  4. The results were subjected to sensitivity tests that involved re-estimating the impacts using three alternative analytic approaches. The findings discussed below are robust to adjustments for multiple comparisons and sensitivity tests unless specified.

The findings discussed below are robust to adjustments for multiple comparisons and sensitivity tests unless specified.

The analysis suggests the following findings regarding the impacts of a scholarship offer (table ES-2):

  • The main models indicate that the Program generated no statistically significant impacts, positive or negative, on student reading or math achievement for the entire impact sample in year 1. One of the three alternative specifications indicated a positive and statistically significant math impact of 3.4 scale score points.
  • No statistically significant achievement impacts were observed for the high-priority subgroup of students who had attended a SINI public school under NCLB before applying to the Program.
  • The Program may have had an impact on math achievement for two subgroups of students with baseline characteristics associated with better academic preparation. The main models suggest that the OSP improved the math achievement of participating students who had not attended a SINI school by 4.7 scale score points and increased the math scores of those with relatively higher test score performance at baseline by 4.3 scale score points. However, these findings should be interpreted with caution, as adjustments for multiple comparisons suggested they may be false discoveries.
  • No significant achievement impacts were observed for other subgroups of participating students, including those with lower test scores at baseline, girls, boys, elementary students, secondary students, or students within each of the individual cohorts that in combination made up the impact sample.

Table ES-2. Year 1 Test Score Differential ITT Regression-Based Impact Estimates

  • The Program had a substantial positive impact on parents' views of school safety but not on students' actual school experiences with dangerous activities. Parents in the treatment group perceived their child's school to be less dangerous (an impact of -0.74 on a 10-point scale) than parents in the control group. Student reports of dangerous incidents in school did not differ systematically between the treatment and control groups.
  • The Program also had an impact on parent satisfaction with their child's school. For example, an additional 19 percent of the parents of students in the treatment group graded their child's school "A" or "B" compared with the parents of control group students.
  • For the most part, student satisfaction with their school was unaffected by the Program. The main exception was for students with lower test score performance at baseline, who on average assigned their schools significantly lower grades if they were in the treatment group.

Additional Findings Regarding Using a Scholarship and Attending a Private School (Non-experimental)

The results described above answer the question "what happened to OSP applicants who were offered a scholarship, whether or not a student used the scholarship to attend a private school?" Estimating the impact of using an OSP scholarship involves statistically adjusting the initial impact results to account for two groups of impact sample students: (1) the about 20 percent who received but failed to take up the scholarship offer, who presumably had zero impact from the Program, and (2) an estimated 4 percent in the control group who never received a scholarship offer but who, by virtue of having a sibling with an OSP scholarship, wound up in a participating private school (what we call "program-induced crossover"). These straightforward statistical adjustments yield what are typically called the "impact-on-the-treated" or IOT results. These adjustments increase the size of the scholarship offer effect estimate, but cannot make a statistically insignificant result significant. Therefore, the adjustments are only applied to results that were statistically significant at the scholarship offer stage of the analysis.

The statistically significant findings regarding the use of a scholarship include:

  • Using a scholarship led to positive impacts on math scores for students from non-SINI schools (6.1 scale score points compared to 4.7 scale score points for the impact of scholarship award) and for students with higher test scores at baseline (5.6 scale score points compared to 4.3 scale score points for the impact of scholarship award). However, adjustments for multiple comparisons indicate both of these findings may be false discoveries.
  • Scholarship use led to an average reduction of nearly a point on the 10-point danger perception index for parents, compared to a -0.74 point impact for the award of a scholarship.
  • Using a scholarship significantly increased parent satisfaction with their child's school. An additional 25 percent of the parents of scholarship users graded their child's school "A" or "B" compared to the parents of control group students, while the difference was 19 percent for the impact of the offer of a scholarship.

Estimating the effect of attending a private school, regardless of whether an OSP scholarship was used, also begins with the original impact results but uses a more complex statistical procedure6. Because this approach deviates somewhat from the overall experimental design of the evaluation, and yields estimates that are less precise, the private schooling results should be interpreted and used with caution. Like those applied to estimate the impact of OSP scholarship use, the private schooling adjustments increase the size of the scholarship offer effect estimate, but cannot make an insignificant result significant. Therefore, the procedure is only applied to results that were statistically significant at the scholarship offer stage of the analysis.

The main private schooling results suggest that

  • Private schooling was associated with higher math achievement for SINI-never students (by 7.8 scale score points) and for students with higher test scores at baseline (by 6.7 scale score points), but both of these findings may be false discoveries due to multiple comparisons. These private schooling differences were larger than were the impacts of scholarship award and scholarship use for SINI-never students (4.7 points and 6.1 points, respectively) and for students with higher test scores at baseline (4.3 points and 5.6 points, respectively).
  • Private schooling is associated with lower parent perceptions of danger and higher parent satisfaction. The average score for private school parents represented 1.14 fewer points or areas of concern on the 10-point school danger index than the average score for public school parents, compared to impacts of -0.74 points for scholarship award and a reduction of one point for scholarship use. Similarly, parents of private school students were 30 percent more likely to grade their child's school an "A" or "B" than were parents of public school students, compared to impact on this measure of 19 percent for scholarship award and 25 percent for scholarship use.

These results can be placed in the context of other RCTs of scholarship programs for low-income students, which suggest no consistent pattern of academic achievement impacts for the first year of program participation. Among such evaluations of four privately funded scholarship programs, one study of the Charlotte, North Carolina, program clearly found statistically significant overall impacts on math and reading for the first year, while one of three analyses of the New York City program found overall impacts on math achievement (Barnard, Frangakis, Hill, and Rubin, 2003; Greene 2000). When African-Americans are considered separately, a group that makes up nearly 90 percent of the OSP impact study sample, two of three analyses of the New York City program suggest there were achievement gains in math for African-American students in some grade levels (Mayer, Peterson, Myers, Tuttle, and Howell, 2002), but studies of the Dayton, Ohio, and earlier District of Columbia programs found no impacts for this group until students were in the program for 2 years (Howell, Wolf, Campbell, and Peterson, 2002). In contrast, all of the randomized controlled trials that measured parent satisfaction and perceptions of school safety found positive impacts similar to those demonstrated by the OSP the first year (Greene, 2000; Howell and Peterson et al., 2002).

The findings here are based on information collected only a year after students applied to the Program and may not reflect the consistent impacts of the OSP over a longer period of time. Families that apply to voucher programs intend for their children to leave their current public schools and, in the case of the OSP, a much higher share of students in the treatment group (91.3 percent) switched schools—mostly from public to private—compared to those in the control group (56.6 percent). The first-year results, therefore, provide an early look at student experiences in what was a transitional year for most of them. Future reports will examine impacts 2 and 3 years after application to the Program, when any short-term effect of students' transition to new schools may have dissipated. The later reports will also consider additional outcome measures, assess the extent to which school characteristics are associated with impacts, and examine how the DC public school system is changing in response to the Program.

6 The scholarship lottery is used as an instrumental variable (IV) to predict whether a student attended private school. Unlike an indicator variable for actual attendance at a private school, the prediction of private school attendance using the scholarship lottery instrument is unbiased because it is the same for all treatment group students (and all control group students) regardless of their individual enrollment decisions.