Skip Navigation
Evaluation of the DC Opportunity Scholarship Program: Impacts After One Year
NCEE 2007-4009
June 2007

The Mandated Evaluation

In addition to establishing the DC Opportunity Scholarship Program, Congress required an independent evaluation that uses ". . . the strongest possible research design for determining the effectiveness" of the Program. The Department of Education's Institute of Education Sciences (IES), responsible for the mandated evaluation, determined that the foundation of the evaluation would be a randomized controlled trial (RCT) that compares outcomes of eligible public school applicants (students and their parents) randomly assigned to receive or not receive a scholarship. An RCT design is widely viewed as the best method for identifying the independent effect of programs on subsequent outcomes and has been used by researchers conducting impact evaluations of privately funded scholarship programs in Charlotte, North Carolina; Dayton, Ohio; New York City; and Washington, DC.3

The RCT design for the OSP evaluation required more applications than scholarships or slots available in private schools, what we call "oversubscription," to permit the random assignment of scholarships through lotteries. However, not all OSP applicants faced conditions for a lottery. The pool of eligible public school applicants in oversubscribed grades included 492 applicants in cohort 1 (spring 2004) and 1,816 applicants in cohort 2 (spring 2005). Of those 2,308 eligible public school applicants who entered lotteries, 1,387 were randomly assigned to receive a scholarship (the "treatment" condition), and 921 were randomly assigned to not receive a scholarship (the "control" condition). The lotteries that generated these assignments took into account the statutory priorities, such that students from SINI schools had the highest probability within their grade bands of being awarded a scholarship, and students from other public schools had a lower probability of being awarded a scholarship. The OSP impact sample group includes the randomly assigned members of the treatment and control groups and comprises 57 percent of all eligible applicants in the first 2 years of Program operation.4

3 RCTs are commonly referred to as the "gold standard" for evaluating educational interventions; when mere chance determines which eligible applicants receive access to school choice, the students who apply but are not admitted make up an ideal "control group" for comparison with the school choice "treatment group." See chapter 3 for more detail on the RCT design and analysis.

4 Students who were already attending a private school when they applied to the OSP are not included in the impact sample, although a lottery was held for those applicants in cohort 1. Also not included in the impact sample are the 851 students who applied in cohort 1 to enter grades K–5, all of whom received scholarships without a lottery because there were more private school slots than applicants at that grade level.