Early Impact Findings

The primary measure of reading achievement for the ERO study is students' scores on the reading comprehension assessment subtest of the GRADE. A secondary measure of students' reading achievement is their scores on the GRADE vocabulary assessment. Following is a summary of the study's early impact findings.

• When analyzed jointly, the ERO programs produced an increase of 0.9 standard score point on the GRADE reading comprehension subtests. This corresponds to an effect size of 0.09 standard deviation and is statistically significant.

The top panel of Table ES.1 shows the impacts on reading comprehension and vocabulary test scores across all 34 participating high schools. The first row in the table shows that, overall, the ERO programs improved reading comprehension test scores by 0.9 standard score point and that this impact is statistically significant (its p-value is less than or equal to 5 percent). Expressed as a proportion of the overall variability of test scores for students in the non- ERO group, this represents an effect size of 0.09 (or 9 percent of the standard deviation on the non-ERO group's test scores).

Figure ES.1 places this impact estimate in the context of the actual and expected change in the ERO students' reading comprehension test scores from the beginning of ninth grade to the end of ninth grade. The bottom section of the bar shows that students in the ERO group achieved an average standard score of 85.9 at the start of their ninth-grade year. This corresponds, approximately, to a grade equivalent of 5.1 (the first month of fifth grade) and indicates an average reading level at the 16th percentile for ninth-grade students nationally. The middle section of the bar shows the estimated growth in test scores experienced by the non-ERO group. This growth of 3.4 standard score points provides the best indication of what the ERO group would have achieved during their ninth-grade year had they not had the opportunity to attend the ERO classes. At the end of the ninth-grade year, therefore, the non-ERO group was estimated to have achieved an average standard score of 89.2, which corresponds to a grade equivalent of 5.9 and an average reading level at the 23rd percentile for ninth-grade students nationally.

The top section of the bar shows the estimated impact of the ERO programs on reading comprehension test scores. At the end of the ninth-grade year, the ERO group was estimated to have achieved an average standard score of 90.1, which corresponds to a grade equivalent of 6.1 and an average reading level at the 25th percentile for ninth-grade students nationally. Thus, the impact of the ERO programs represents a 26 percent improvement over and above what the ERO group would have achieved if they had not had the opportunity to attend the ERO classes.³

The solid line at the top of Figure ES.1 shows the national average (100 standard score points) for students at the end of ninth grade, in the spring. Students scoring at this level are considered to be reading at grade level. Despite the program impact, therefore, the ERO group's reading comprehension scores still lagged nearly 10 points below the national average. In fact, almost 90 percent of the students in the ERO group had reading comprehension scores that were below grade level at the end of ninth grade. Hence, 76 percent of students who participated in the ERO classes would still be eligible for the programs because they had scored more than two years below grade level at the end of their ninth-grade year.

• Although neither program-specific impact is statistically significant, estimated impacts for schools using the Reading Apprenticeship Academic Literacy program and for schools using Xtreme Reading are 0.9 standard score point.

Table ES.1 shows that the impacts on reading comprehension for both Reading Apprenticeship and Xtreme Reading are of similar magnitude to that found for the full sample of schools in the study. Neither of these estimates is statistically significant, however.

The ERO Student Follow-up Survey included questions about students' reading behavior. The impact analysis focused on three measures that were developed from these questions: the amount of reading students do for school, the amount of reading students do for non-school purposes, and students' use of reflective reading strategies. While the ERO programs produced some changes in these reading behaviors (both positive and negative), none of the estimated impacts is statistically significant.

Top

---

³ This was calculated by dividing the impact (0.9 standard score point) by the average improvement of the non-ERO group (3.4 standard score points).