Skip Navigation

National Center for Education Evaluation and Regional Assistance


Evaluation of the DC Opportunity Scholarship Program:

NCEE 2009-4050
March 2009

Executive Summary

The District of Columbia School Choice Incentive Act of 2003, passed by Congress in January 2004, established the first federally funded, private school voucher program in the United States. As part of this legislation, Congress mandated a rigorous evaluation of the impacts of the Program, now called the DC Opportunity Scholarship Program (OSP). This report presents findings from the evaluation of the impacts 3 years after families who applied were given the option to move from a public school to a participating private school of their choice.

The evaluation is based on a randomized controlled trial design that compares the outcomes of eligible applicants randomly assigned to receive (treatment group) or not receive (control group) a scholarship through a series of lotteries. The main findings of the evaluation so far include:

  • After 3 years, there was a statistically significant positive impact on reading test scores, but not math test scores. Overall, those offered a scholarship were performing at statistically higher levels in reading—equivalent to 3.1 months of additional learning—but at similar levels in math compared to students not offered a scholarship (table 3). Analysis in prior years indicated no significant impacts overall on either reading or math achievement.
  • The OSP had a positive impact overall on parents' reports of school satisfaction and safety (figures 3 and 4), but not on students' reports (figures 3 and 4). Parents were more satisfied with their child's school (as measured by the percentage giving the school a grade of A or B) and viewed their child's school as safer and more orderly if the child was offered a scholarship. Students had a different view of their schools than did their parents. Reports of safety and school climate were comparable for students in the treatment and control groups. Overall, student satisfaction was unaffected by the Program.
  • This same pattern of findings holds when the analysis is conducted to determine the impact of using a scholarship rather than being offered a scholarship. Fourteen percent of students in our impact sample who were randomly assigned by lottery to receive a scholarship and who responded to year 3 data collection chose not to use their scholarship at any point over the 3-year period after applying to the Program.1 We use a common statistical technique to take those "never users" into account; it assumes that the students had zero impact from the OSP, but it does not change the statistical significance of the original impact estimates. Therefore, the positive impacts on reading achievement, parent views of school safety and climate, and parent views of satisfaction all increase in size, and there remains no impact on math achievement and no overall impact on students' perceptions of school safety and climate or satisfaction from using an OSP scholarship.
  • The OSP improved reading achievement for 5 of the 10 subgroups examined.2 Being offered or using a scholarship led to higher reading test scores for participants who applied from schools that were not classified as "schools in need of improvement" (non-SINI). There were also positive impacts for students who applied to the Program with relatively higher levels of academic performance, female students, students entering grades K-8 at the time of application, and students from the first cohort of applicants. These impacts translate into 1/3 to 2 years of additional learning growth. However, the positive subgroup reading impacts for female students and the first cohort of applicants should be interpreted with caution, as reliability tests suggest that they could be false discoveries.
  • No achievement impacts were observed for five other subgroups of students, including those who entered the Program with relative academic disadvantage. Subgroups of students who applied from SINI schools (designated by Congress as the highest priority group for the Program) or were in the lower third of the test score distribution among applicants did not demonstrate significant impacts on reading test scores if they were offered or used a scholarship. In addition, male students, those entering high school grades upon application, and those in application cohort 2 showed no significant impacts in either reading or math after 3 years.

Top

1 This 14 percent "never user" rate among year 3 respondents in the impact sample differs from the 25 percent "never user" rate for the impact sample as a whole (Figure 1) because scholarship "never users" in the impact sample responded to year 3 data collection events at lower rates than did scholarship "ever users."
2 The subgroups that are analyzed in this study were designated prior to the collection and analysis of data and are of particular policy interest based on the Program statute and education policy literature. The subgroups are: (1) whether students attended a school designated as in need of improvement (SINI) under the No Child Left Behind Act prior to application to the Program — students were either attending a SINI-ever or SINI-never school; (2) whether students were relatively lower performing or relatively higher performing at baseline — students were either in the bottom one-third or the top two-thirds of the test score distribution; (3) student gender; (4) whether students were entering grades K-8 or 9- 12 at the time of application; and (5) whether students were in application cohort 1 (applied in 2004) or application cohort 2 (applied in 2005).