Appendix A: Congressionally Mandated Evaluation

Research Questions

Based on guidance in the statute, the research team plans to conduct a comprehensive and rigorous RCT evaluation of the impact of the OSP on participating students and families. Specifically, the evaluation will address the following research questions:

1. What is the impact of the Program on student academic achievement? The law places high priority on examining whether the Program—the availability and offer of scholarships—improves the academic achievement of eligible students. This question can be addressed most rigorously by comparing the academic achievement of student applicants randomly assigned by a lottery to receive and not receive scholarships.
2. What is the impact of attending private versus public schools? Because some students offered scholarships will choose not to use them, the research team will use accepted statistical methods to examine the effects for students who take the scholarship offer and successfully enroll in a private school.
3. What is the impact of the Program on other student measures? The law calls for examining other indicators of student and school success, including persistence, retention, graduation, and, if possible, college enrollment. In addition, the legislation requires the evaluation to assess the school safety of students who receive the scholarships relative to those who did not receive scholarships.
4. What effect does the Program have on student and parent satisfaction with the educational options available in the District and with children’s actual school experiences? A key desired outcome of scholarship programs is an increase in both the school choices possible and parents’ and students’ satisfaction with the choices they have made. These issues will be examined by comparing the satisfaction and reasons for applying to the DC Opportunity Scholarship Program among applicants assigned by lottery to receive scholarships and those assigned to not receive scholarships.
5. To what extent is the Program influencing public schools and expanding choice options for parents in Washington, DC? Scholarship programs have been hypothesized to affect not only the students who receive the scholarships but also the broader population of public schools and students. Theory suggests that these broader outcomes could occur if a significant number of students move from public to private schools. The public schools might experience a reduction in per-student funding that affects their offerings, a change in average student performance, or they may respond to the competition for students by changing curricula, adopting new themes or missions, and modifying existing policies and practices to make the public schools more attractive to students with schooling options. Choice programs might also affect the larger population of private schools, beyond those in which the programs’ participants are currently enrolled; if choice programs are successful, additional private schools may choose to participate, new schools may be established to meet enrollment demand, or existing schools might expand capacity. These issues will be explored with descriptive analyses based primarily on school surveys.