Study sites differ from each other in ways that could potentially influence the effectiveness of Reading First. For example, sites differ in terms of the length of time since date of Reading First grant award, levels of Reading First funding per student, and prior levels of reading performance. Consequently, average impacts for the full study sample might mask important differences that exist over time and/or across sites. The study explored this possibility by examining the pattern of impacts over time for two groups of study sites. The first group consists of the eight "late award" sites that received Reading First grants between January and August 2004. As of May 2006, these sites had been receiving Reading First funds for an average of approximately two years. The second group consists of the 10 "early award" sites that received Reading First grants between April and December 2003. As of May 2006, these sites had been receiving Reading First funds for an average of approximately three years, although data from the study are available only for the last two years. Study findings indicate that:
- The impacts of Reading First on classroom instruction and student reading comprehension have not changed consistently over time. Exhibit ES.4 shows estimated impacts for the two years that data are available for late award and early award sites, respectively. For both groups of sites, estimates of program impacts on reading comprehension and classroom instruction vary from year to year (across columns). However, this variation exhibits no consistent pattern and is not statistically significant. These findings do not suggest that program impacts increased or decreased with program maturity.
- The estimated impacts of Reading First were consistently positive for late award sites and mixed for early award sites. Exhibit ES.5 presents estimated impacts for the two groups of sites that are averaged over the two years for which data are available. It indicates that, for grades one and two in late award sites, Reading First produced positive and statistically significant increases both in teachers' instruction in the five dimensions and in students' reading comprehension. Impacts on third grade reading comprehension were not statistically significant for late award sites, though the direction of the (not significant) estimated impact was positive. None of the impact estimates presented in Exhibit ES.5 are statistically significant for early award sites. The (not significant) estimated impacts on teachers' instruction were positive, and the (not significant) estimated impacts on student reading comprehension were negative. Differences in impacts on reading comprehension test scores between early and late award sites were statistically significant for grades two and three, and not statistically significant for grade one. Differences in impacts on instruction in the five dimensions between early and late award sites were not statistically significant.
- It is not possible to determine which of numerous differences between early award sites and late award sites may have caused observed differences in Reading First impacts, only some of which were statistically significant. The average per K-3 student Reading First funding was higher in late award sites than early award sites ($574 versus $432 per student). Although the study did not begin to collect data until after early award sites began to implement Reading First, it appears that the benchmarks of comparison for student reading comprehension were lower for late award sites. Thus, late award sites may have had more room to increase reading comprehension skills. Any or all of these differences, plus others not measured, could have produced the impact differences observed.
Exhibit ES-4. Estimated Impacts on Reading Comprehension and Minutes in the Five Dimensions, by Implementation Year, Calendar Year, and Award Status
Exhibit ES-5. Estimated Impacts on Key Outcomes for Early and Late Award Sites, by Grade