Skip Navigation
Evaluation of the DC Opportunity Scholarship Program

April 2005

Scholarship and Placement Lotteries and Initial Use of Scholarships Awarded

The program statute requires that scholarship recipients be randomly selected (e.g., by lottery), if the program or specific schools are "oversubscribed"—that is, have more demand for them than can be accommodated. The law also details congressional priorities to guide the award of scholarships and scarce seats to eligible applicants: (1) students attending a public school designated as in need of improvement (SINI) under the No Child Left Behind Act at the time of application to the program, and (2) families that lack the resources to take advantage of the educational choices available to them.

A total of 79 eligible applicants (4 percent) were from one of the 15 SINI-designated schools in spring 2004 and were, therefore, given the highest priority in the lotteries.1 An additional 1,251 eligible applicants (68 percent) were attending non-SINI public schools and were assigned the second-highest priority in the lotteries. The 518 eligible applicants (28 percent) from private schools were given the lowest priority, because they were considered to meet neither of the congressional priorities. These priority groups were used both to award scholarships and, later, to place scholarship recipients in the participating private schools of their choice.

The Scholarship Lottery

The first lottery was to distribute scholarships to eligible applicants. For public school applicants, each student’s probability of obtaining a scholarship was dependent not only on his or her membership in a priority group but also on the availability of new private school seats at various grade levels. The new seats in participating private schools were highly concentrated in the K-5 elementary grades, meaning there were more available seats in those grades than there were eligible applicants (Figure ES-1).

Thus, in the lottery, K-5 students would receive scholarships automatically. The middle school grades (6-8) were modestly oversubscribed, and the high school grades (9-12) were severely oversubscribed. Only in those grades was there random assignment as part of the lottery. Since eligible private school applicants already held slots in their private schools, they were not constrained by slot availability in the same way as public school applicants.

Accounting for both the statutory priorities and the slot constraints within the grade-level bands, scholarship award probabilities were assigned to the various groups of eligible applicants and a custom-designed computer program awarded scholarships to students within each group (Figure ES-2).

A total of 1,366 scholarships were awarded in June 2004, including the following:

  • All 79 applicants from SINI-designated schools.2

  • All 772 non-SINI public school applicants in K to 5.

  • 255 (76 percent) of the non-SINI public school applicants in grades 6 to 8 and 44 (28 percent) of them in grades 9 to 12.

  • A total of 216 (43 percent) of applicants already attending private schools, with probabilities considerably lower (55 percent for those in grades K to 5, 42 percent for grades 6 to 8, and 17 percent for grades 9 to 12) than public school applicants in all three grade bands.

In sum, the scholarship lottery produced two groups of students for purposes of meeting statutory requirements for the DC Choice Opportunity Scholarship program evaluation. The 1,366 scholarship recipients will be the subject of annual performance reporting and comparison to DCPS nonapplicants. The 492 public school applicants who were entering grades subject to random assignment (grades 6-12) will contribute to the annual impact analysis: these include 299 students assigned to the treatment group (and also included in the performance reporting sample) and 193 students assigned to the control group. The 289 private school applicants who were not awarded scholarships belong to neither group. Since they previously attended and presumably will continue to attend private school using resources outside of the scholarship program, following these students after baseline would not contribute meaningfully to the evaluation. To conserve resources, this group of initial applicants will not be part of the evaluation going forward.

However, the impact sample in the first year of the program is not, on its own, large enough for the evaluation to reliably draw conclusions about any differences in achievement outcomes that might be expected from an intervention of this kind.3 Instead, the treatment and control groups from the first year lottery will be combined with those from the lottery for second year applicants, expected in April 2005, to provide a sufficient sample for the rigorous evaluation of program impacts.


Placement Lottery and Follow Up

After being notified of their scholarship offer, families were required to meet with officials at participating private schools and obtain conditional acceptance to the schools that they wanted their children to attend. Parents then submitted school preference forms to indicate and rank the top four private schools of their choice. These forms were used to place students, through a combination of a custom-designed computer placement lottery and followup case-by-case placements by WSF. There were a total of 1,366 scholarship winners:

  • A total of 1,040 were successfully placed, with the overwhelming majority of students placed in their most-preferred school.

  • The remaining 326 scholarship recipients did not complete the school search process and, therefore, could not be assigned a placement in a participating private school.

Of the 1,040 who were placed, 1,027 had matriculated at their preferred private school by September 10, 2004. This represents an overall initial scholarship usage rate of 75 percent. The usage rate for the impact sample’s treatment group is lower—62 percent—because that group excludes students in grades K-5 and those who were already attending private schools when they applied to the program, subgroups that have significantly higher rates of scholarship use than do public school students in the middle and high school grades.

Top

1 While there were 79 applicants from the 15 schools designated as SINI in 2003, in advance of the first year lottery in May 2004, DCPS designated an additional 73 schools as in need of improvement in August 2004, two months after the scholarship lottery (See http://silicon.k12.dc.us/NCLB/index.asp, accessed February 15, 2004). A total of 535 program applicants came from 2003-designated or 2004-designated SINI schools, representing 29 percent of all program applicants this first year. Of these students, 433 (81 percent) were awarded scholarships. Twice as many Opportunity Scholarships were awarded to students applying from 2003- or 2004-designated SINI schools than were awarded to students applying from private schools.
2 Because they represented such a small share of the overall applicant pool the first year and there was little possibility of separately analyzing impacts for this subgroup, all SINI applicants were awarded a scholarship. However, with more SINI schools identified for the next year, it is likely that a much higher number of students in this priority group will apply to the program in the future. Therefore, in succeeding years the group will be given highest priority but be part of the random assignment process that will enable that important category of students to be included in the rigorous impact evaluation.
3 Evaluation sample size needs are calculated using statistical power analysis. It is an important part of evaluation design and demonstrates how well the design will be able to distinguish real impacts from chance differences between groups. The analysis takes into consideration such factors as the size of the impact expected, the proportion of students anticipated to be assigned to the treatment and control groups, and the likely attrition from the study’s data collection. The power analysis conducted for the DC Opportunity Scholarship Program suggests a target of 1,240 randomly assigned students, with slightly more than 800 students assigned to the treatment group and slightly more than 400 assigned to the control group, so long as the test score impacts in the outcomes in future years are at least moderately large (Appendix A).