Executive Summary
This report presents findings from the Enhanced Reading Opportunities (ERO) study —
a demonstration and rigorous evaluation of two supplemental literacy programs that aim to improve
the reading comprehension skills and school performance of struggling ninth-grade readers.
The U.S. Department of Education’s (ED) Office of Elementary and Secondary Education
(OESE)1 is funding the implementation of these programs, and its Institute of Education
Sciences (IES) is responsible for oversight of the evaluation. MDRC — a nonprofit, nonpartisan
education and social policy research organization — is conducting the evaluation in partnership
with the American Institutes for Research (AIR) and Survey Research Management (SRM).
The present report — the second of three — focuses on the second of two cohorts of
ninth-grade students to participate in the study and discusses the impact that the two interventions
had on these students’ reading comprehension skills through the end of their ninth-grade
year. The report also describes the implementation of the programs during the second year of
the study and provides an assessment of the overall fidelity with which the participating schools
adhered to the program design as specified by the developers. While this report focuses primarily
on implementation and impacts in the second year of the study, comparisons between the first
and second year of the study are also provided.2 The key findings discussed in the report include
the following:
- On average, across the 34 participating high schools, the supplemental
literacy programs improved student reading comprehension test scores
by 0.08 standard deviation. This represents a statistically significant improvement
in students’ reading comprehension (p-value = 0.042).
- Seventy-seven percent of the students who enrolled in the ERO classes in
the second year of the study were still reading at two or more years below
grade level at the end of ninth grade, relative to the expected reading
achievement of a nationally representative sample of ninth-grade
students.3 One of the two interventions — Reading Apprenticeship Academic Literacy (RAAL) — had a positive and statistically significant impact on reading comprehension test scores (0.14 standard deviation;
p-value = 0.015). Although not statistically significant, a positive impact
on reading comprehension (0.02 standard deviation) was also produced
by the other intervention, Xtreme Reading. The difference in impacts
between the two programs is not statistically significant, and thus it cannot
be concluded that RAAL had a different effect on reading comprehension
than Xtreme Reading.4
- The overall impact of the ERO programs on reading comprehension test
scores in the second year of implementation (0.08 standard deviation) is
not statistically different from their impact in the first year of implementation
(0.09 standard deviation), nor is each intervention’s impact in the
second year of implementation statistically different from its impact in
the first year.
- The implementation fidelity of the ERO programs was more highly
rated in the second year of the study than in the first year. In comparison
with the first year, a greater number of schools in the second year of
the study were deemed to have programs that were well aligned with the
program developers’ specifications for implementation fidelity (26
schools in the second year, compared with 16 schools in the first year),
and fewer schools were considered to be poorly aligned (one school in
the second year, compared with 10 schools in the first year).
Top
1 The implementation was initially funded by the Office of Vocational and Adult Education (OVAE), but this role was later transferred to OESE.
2 James J. Kemple, William Corrin, Elizabeth Nelson, Terry Salinger, Suzannah Herrmann, and Kathryn Drummond, The Enhanced Reading Opportunities Study: Early Impacts and Implementation Findings, NCEE 2008-4015 (Washington, DC:, U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, 2008).
3 Forty percent of ninth-graders nationally would be expected to score at two or more years below grade level on the same assessment.
4 It is important to note that the ERO study is an evaluation of a class of reading interventions, as
represented by Xtreme Reading and RAAL, as well as an evaluation of each of these two programs separately.
The purpose of the study is not to test the differential impact of these two interventions; while Xtreme Reading
and RAAL do differ in some respects, they are both full-year supplemental literacy courses targeted at struggling
adolescent readers that share many common principles, and hence there was no prior expectation that
they would produce substantially different impacts. As noted below, the design of the study is such that programs
are randomized to schools; however, the purpose of this randomization was to ensure that each program
developer was assigned a fair draw of schools in which to implement its program, rather than to test for a differential
impact between the two interventions. By this token, the statistical model chosen for the impact analysis
does not utilize the school-level randomization feature of the research design; nor is the sample size large
enough to detect policy-relevant differences in impacts across the two programs. Because Xtreme Reading and
RAAL represent the same type of intervention, this study was designed to test their joint or overall impact.
Statistical tests were used to confirm that the difference in impacts between the two programs is not statistically
significant and, hence, that it is indeed appropriate to pool together the two program-specific impact estimates;
these statistical tests are not appropriate for making inferences about the true difference in impacts between
the two interventions.