Skip Navigation

National Center for Education Evaluation and Regional Assistance

The Enhanced Reading Opportunities Study: Findings from the Second Year of Implementation

NCES 2009-4036
November 2008

Executive Summary

This report presents findings from the Enhanced Reading Opportunities (ERO) study — a demonstration and rigorous evaluation of two supplemental literacy programs that aim to improve the reading comprehension skills and school performance of struggling ninth-grade readers. The U.S. Department of Educationís (ED) Office of Elementary and Secondary Education (OESE)1 is funding the implementation of these programs, and its Institute of Education Sciences (IES) is responsible for oversight of the evaluation. MDRC — a nonprofit, nonpartisan education and social policy research organization — is conducting the evaluation in partnership with the American Institutes for Research (AIR) and Survey Research Management (SRM).

The present report — the second of three — focuses on the second of two cohorts of ninth-grade students to participate in the study and discusses the impact that the two interventions had on these studentsí reading comprehension skills through the end of their ninth-grade year. The report also describes the implementation of the programs during the second year of the study and provides an assessment of the overall fidelity with which the participating schools adhered to the program design as specified by the developers. While this report focuses primarily on implementation and impacts in the second year of the study, comparisons between the first and second year of the study are also provided.2 The key findings discussed in the report include the following:

  • On average, across the 34 participating high schools, the supplemental literacy programs improved student reading comprehension test scores by 0.08 standard deviation. This represents a statistically significant improvement in studentsí reading comprehension (p-value = 0.042).
  • Seventy-seven percent of the students who enrolled in the ERO classes in the second year of the study were still reading at two or more years below grade level at the end of ninth grade, relative to the expected reading achievement of a nationally representative sample of ninth-grade students.3 One of the two interventions — Reading Apprenticeship Academic Literacy (RAAL) — had a positive and statistically significant impact on reading comprehension test scores (0.14 standard deviation; p-value = 0.015). Although not statistically significant, a positive impact on reading comprehension (0.02 standard deviation) was also produced by the other intervention, Xtreme Reading. The difference in impacts between the two programs is not statistically significant, and thus it cannot be concluded that RAAL had a different effect on reading comprehension than Xtreme Reading.4
  • The overall impact of the ERO programs on reading comprehension test scores in the second year of implementation (0.08 standard deviation) is not statistically different from their impact in the first year of implementation (0.09 standard deviation), nor is each interventionís impact in the second year of implementation statistically different from its impact in the first year.
  • The implementation fidelity of the ERO programs was more highly rated in the second year of the study than in the first year. In comparison with the first year, a greater number of schools in the second year of the study were deemed to have programs that were well aligned with the program developersí specifications for implementation fidelity (26 schools in the second year, compared with 16 schools in the first year), and fewer schools were considered to be poorly aligned (one school in the second year, compared with 10 schools in the first year).


1 The implementation was initially funded by the Office of Vocational and Adult Education (OVAE), but this role was later transferred to OESE.
2 James J. Kemple, William Corrin, Elizabeth Nelson, Terry Salinger, Suzannah Herrmann, and Kathryn Drummond, The Enhanced Reading Opportunities Study: Early Impacts and Implementation Findings, NCEE 2008-4015 (Washington, DC:, U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, 2008).
3 Forty percent of ninth-graders nationally would be expected to score at two or more years below grade level on the same assessment.
4 It is important to note that the ERO study is an evaluation of a class of reading interventions, as represented by Xtreme Reading and RAAL, as well as an evaluation of each of these two programs separately. The purpose of the study is not to test the differential impact of these two interventions; while Xtreme Reading and RAAL do differ in some respects, they are both full-year supplemental literacy courses targeted at struggling adolescent readers that share many common principles, and hence there was no prior expectation that they would produce substantially different impacts. As noted below, the design of the study is such that programs are randomized to schools; however, the purpose of this randomization was to ensure that each program developer was assigned a fair draw of schools in which to implement its program, rather than to test for a differential impact between the two interventions. By this token, the statistical model chosen for the impact analysis does not utilize the school-level randomization feature of the research design; nor is the sample size large enough to detect policy-relevant differences in impacts across the two programs. Because Xtreme Reading and RAAL represent the same type of intervention, this study was designed to test their joint or overall impact. Statistical tests were used to confirm that the difference in impacts between the two programs is not statistically significant and, hence, that it is indeed appropriate to pool together the two program-specific impact estimates; these statistical tests are not appropriate for making inferences about the true difference in impacts between the two interventions.