First-Year Implementation Challenges and Early Impacts

The first-year start-up experiences in 19 of the 34 participating high schools were particularly problematic either because of poorly aligned implementation fidelity or because of especially long delays in enrolling students in their ERO classes. Of these, seven high schools experienced poorly aligned implementation, even though they were able to begin the classes within six weeks of the start of the school year, and nine high schools experienced a start-up delay of more than six weeks, even though the implementation of their ERO programs ended up being classified as at least moderately aligned with their program models. The remaining three high schools experienced both poorly aligned implementation and a start-up delay of more than six weeks. The presence of these implementation challenges in 19 of the high schools raises questions about whether the ERO programs had stronger impacts for the 15 high schools that were able to begin classes within six weeks of the start of the school year and where implementation was classified as moderately or well aligned with the program models.

Table ES.2 shows the impacts on reading test scores for the two groups of schools defined by their first-year start-up experiences. The top panel of the table shows the impacts for the 15 schools that operated their programs for at least seven and a half months and reached a limited or adequate level of implementation fidelity on both the classroom learning environment and the comprehension instruction dimensions. These ERO programs produced positive and statistically significant impacts on reading comprehension test scores. The schools with a stronger start-up produced an increase of 1.8 standard score points in reading comprehension. This is equivalent to an effect size of 0.17 standard deviation and is statistically significant.

The bottom panel of Table ES.2 presents estimated impacts on reading comprehension test scores for ERO programs in schools where implementation fidelity was found to be inadequate or where the programs operated for seven and a half months or less in the first year. The difference is not statistically significant. The difference between the impact for the stronger start-up schools and the weak start-up schools is 1.6 standard score points and an effect size of 0.16. This difference in impacts is statistically significant, indicating that there is a systematic difference in impacts across these two groups of schools.

It is important to note that the analyses just discussed are exploratory and are not able to establish causal links between these early implementation ross the sites. There are other school characteristics and implementation factors that may also be associated with variation in estimated impacts. As an exploratory analysis, it is also not appropriate to extrapolate from these findings to determine the impact of the ERO programs in the second year of the project.

Top