Impact Findings
The GRADE assessment was used to measure students’ reading achievement prior to random assignment (at "baseline") and then again in the spring at the end of their ninth-grade year (at "follow-up"). The GRADE is a norm-referenced, research-based reading assessment that is used widely to measure performance and track the growth of an individual student and groups of students. Because the two ERO programs focus primarily on helping students use contextual clues to understand the meaning of words, the reading comprehension subtest of the GRADE is the primary measure of reading achievement in this study, while the GRADE vocabulary subtest is a secondary indicator of the programs’ effectiveness. Performance levels and impacts on both subtests are presented in standard score units; students with a standard score of 100 points are considered to be reading at grade level.9
Following is a summary of the study’s impact findings.
- When analyzed jointly, the ERO programs produced an increase of 0.8 standard score point on the GRADE reading comprehension subtests. This corresponds to an effect size of 0.08 standard deviation and is statistically significant. The overall impact of the programs in the second year of implementation is not statistically different from their overall impact in the first year of implementation (0.09 standard deviation).
The top panel of Table ES.1 shows the impacts on spring follow-up reading comprehension and vocabulary test scores across all 34 participating high schools in the second year of the study. The first row of data in the table shows that, on average, the reading comprehension test scores of students in the ERO group are 0.8 standard score point higher than the scores of students in the non-ERO group, which represents a statistically significant impact (its p-value is less than or equal to 5 percent).10 Expressed as a proportion of the overall variability of test scores for students in the non-ERO group, this estimated impact represents an effect size of 0.08 (or 8 percent of the standard deviation of the non-ERO group’s test scores).
Figure ES.1 places this impact estimate in the context of the actual and expected change in the ERO students’ reading comprehension test scores on the GRADE from the beginning of ninth grade to the end of ninth grade. The bottom section of the bar shows that students in the ERO group achieved an average standard score of 84.6 at the start of their ninth-grade year. This corresponds, approximately, to a grade equivalent of 4.9 (the last month of fourth grade) and indicates an average reading level at the 14th percentile for ninth-grade students nationally.
The middle section of the bar shows the estimated growth in test scores experienced by the non-ERO group. At the end of the ninth-grade year, the non-ERO group was estimated to have achieved an average standard score of 89.3, which corresponds to a grade equivalent of 6.0 and an average reading level at the 23rd percentile for ninth-grade students nationally. This growth of 4.7 standard score points for the non-ERO group provides the best indication of what the ERO group would have achieved during their ninth-grade year had they not had the opportunity to attend the ERO classes.
The top section of the bar shows the estimated impact of the ERO programs on reading comprehension test scores. At the end of the ninth-grade year, the ERO group achieved an average standard score of 90.1, which corresponds to a grade equivalent of 6.1 and an average reading level at the 25th percentile for ninth-grade students nationally. This means that the ERO group experienced a growth of 5.5 points in their reading comprehension skills over the course of ninth grade, which is 0.8 point higher than the growth achieved by the non-ERO group. Thus, the impact of the ERO programs (0.8 standard score point) represents a 17 percent improvement over and above the growth that the ERO group would have experienced if they had not had the opportunity to attend the ERO classes (4.7 points).11
The solid line at the top of Figure ES.1 shows the national average (100 standard score points) for students at the end of ninth grade, in the spring. Students scoring at this level are considered to be reading at grade level. Thus, the ERO group’s reading comprehension scores still lagged nearly 10 points below the national average. In fact, 77 percent of students who participated in the ERO classes scored two or more years below grade level at the end of their ninth-grade year,12 which means that they would still be eligible for the ERO programs were these programs again made available to them.13
- The RAAL program increased students' reading comprehension test scores by a statistically significant amount (0.14 standard deviation). Although not statistically significant, an impact of 0.2 standard score point on reading comprehension (0.02 standard deviation) was produced by the Xtreme Reading program. The difference in impacts between the two programs is not statistically significant, and thus it cannot be concluded that RAAL had a different effect than Xtreme Reading. Nor is there a statistically significant difference between each program’s impact in the second year of implementation and its impact in the first year of implementation.
The ERO student follow-up survey was administered to students at the same time as the follow-up GRADE assessment and includes additional information on students’ reading behaviors and attitudes. Responses to the follow-up survey were used to derive measures for three reading behaviors that are intended to be affected by the ERO programs: the number of times during the prior month that a student read different types of text in school or for homework, the number of times during the prior month that a student read different types of text outside of school, and students’ reported use of the reading strategies and techniques that the ERO programs try to teach. The overall impact of the programs on students' reading behaviors is not statistically significant.14
9 Based on the national norms used to calculate these scores, a standard score of 100 on the GRADE reading comprehension or vocabulary test is average for a representative group of students at the end of their ninthgrade year. The standard deviation of the standard score for both tests is 15.
10 The impact estimates in Table ES.1 are regression-adjusted using ordinary least squares (OLS), controlling for blocking of random assignment by school and for random differences between the ERO and non-ERO groups in their baseline reading comprehension test scores and age at random assignment. The values in the column labeled "ERO Group" are the observed means for students randomly assigned to the ERO group. The "Non-ERO Group" values in the next column are the regression-adjusted means for students randomly assigned to the non-ERO group, using the observed mean covariate values for the ERO group as the basis for the adjustment.
11 The value of 17 percent was calculated by dividing the impact (0.8 standard score point) by the average improvement of the non-ERO group (4.7 standard score points).
12 Forty percent of ninth-graders nationally would be expected to score two years or more below grade level on the GRADE administered in the spring of ninth grade.
13 Furthermore, 87 percent of the students in the ERO group had reading comprehension scores that were below grade level at the end of ninth grade.
14 The analysis also examines the extent to which impacts on reading comprehension test scores vary across schools. The impact estimates for each school range from a negative impact of 3.7 standard score points to a positive impact of 6.2 standard score points. However, the variation in observed school-level impacts is not statistically significant, indicating that the observed school-to-school variation in impacts may be due to estimation error and may not truly vary across schools.