Findings for the Reading Program

Again, the reading findings presented in this report pertain to the 12 centers that partici-pated in two years of program operations and data collection.

Implementation of the Enhanced After-School Reading Program

The enhanced reading program was staffed as intended and offered the intended amount of instruction in both years of program operations. Each center was expected to hire certified teachers and to operate with 10 students per instructor. In the second year, for example, all instructors were certified teachers, and the programs operated with the intended small groups of students — on average, in the second year, nine students attended per instructor. The goal was to offer the program for approximately 180 minutes per week, and average offerings were 177 minutes in the first year and 175 minutes in the second. Instructors were trained by SFA staff at the beginning of the year and were provided ongoing assistance.8 They also received paid preparation time.

However, in both years of the study, instructors found it challenging to maintain the in-tended pace of instruction. In the first year of the study, 79 percent of instructors reported that it was consistently or sometimes difficult to include all aspects of the reading program and maintain the intended pace of the daily lesson plan. In the second year of the study, half of the responding district coordinators reported that pacing continued to be a problem.

Classroom observations conducted by district coordinators were used to assess the fidelity with which instructors implemented the enhanced reading program. In the classes with students at the first-and second-grade reading levels (in Adventure Island, students are grouped by their initial reading level, not by grade), average fidelity scores did not statistically differ across the first and second years of implementation;⁹ in the classes with students reading above the second-grade level, average scores were lower in the second year, by a statistically signifi-cant amount (p-value = 0.00).¹⁰ It was also found that, in any given year, implementation of the program lacked consistency, as indicated by variation in the number of program components implemented by teachers.¹¹ In particular, in the second implementation year, returning teachers in both the lower and upper levels of the program had statistically significantly higher imple-mentation fidelity scores than teachers who were new to the program (p-value = 0.00).

Impacts from Offering One Year of the Enhanced Reading Program

This analysis focuses on the impact of one year of enrollment in the enhanced reading program on student outcomes.¹² The difference between the background characteristics of students in the enhanced and regular program groups, both in Cohort 1 and Cohort 2, was greater than what would be predicted by chance, especially as related to baseline reading achievement test scores and household composition.¹³ Measures of student characteristics (including students’ baseline test score) were included in the impact model to control for observed differences between the two program groups at baseline. Sensitivity analyses were conducted to gauge whether these covariates adequately control for baseline differences between students in the two program groups. These tests confirm that controlling for students’ baseline characteristics — and particularly their pretest scores — produces internally valid estimates of the impact of the enhanced program.

On average, students in the enhanced program reading group in Cohort 1 received 54 more hours of academic instruction in reading during the school year than students in the regular program group. This difference — which is statistically significant (p-value = 0.00) — represents an estimated 22 percent increase in total reading instruction over and above what is received by these students during the regular school day. In Cohort 2, enhanced program students received 56 more hours — also a statistically significantly greater amount of time (p-value = 0.00) than received by those in the regular program group, and an estimated 23 percent increase in total reading instruction. And the net difference in added hours of instructional reading between implementation years is not statistically significant (p-value = 0.63).

One year of enrollment in the enhanced after-school reading program did not have a statistically significant impact on students’ reading achievement (as measured by SAT 10 total reading scores), whether in the first or second year of implementation. It also did not have a significant impact on students’ performance on locally administered standardized reading tests, nor did it produce impacts on the DIBELS measures of fluency or on regular-school-day teacher reports of academic behaviors (homework completion, attentiveness in class, and disruptiveness in class).

Impacts from Offering Two Years of the Enhanced Reading Program

The impact of offering students the opportunity to participate in the enhanced reading program for two consecutive years is estimated using the two-year sample in the same way as for the math sample, by comparing the outcomes of students who were randomly assigned to either the enhanced after-school program or the regular after-school program for two consecu-tive school years.¹⁴ The difference between the background characteristics of students in the enhanced and regular program groups in the two-year sample was greater than what would be predicted by chance, especially related to baseline reading achievement test scores and house-hold composition.¹⁵ Measures of student characteristics (including students’ baseline test scores) were included in the impact model to control for observed differences between the two program groups at baseline. Sensitivity analyses were conducted to gauge whether these covariates adequately control for baseline differences between students in the two program groups. These tests confirm that controlling for students’ baseline characteristics — and particularly their pre-test score — produces internally valid estimates of the impact of the enhanced program groups at baseline. Sensitivity analyses were conducted to gauge whether these covariates adequately control for baseline differences between students in the two program groups. These tests confirm that controlling for students’ baseline characteristics — and particularly their pre-test score — produces internally valid estimates of the impact of the enhanced program.

The estimated impact of offering students the opportunity to enroll in the enhanced after-school program for two consecutive years is negative and statistically significant (-5.6 scaled score points on SAT 10 total reading scores; p-value = 0.04). To place these results into context, the estimated impact on these students of their first year of program enrollment (-3.6 points) was not statistically significant. And the estimated impact of assigning students to two years of enhanced services does not statistically differ from the impact on these students of their first year of access to the program (p-value = 0.46). Hence, while it can be said that being assigned to two years of enhanced services produces significantly fewer gains on test scores, it cannot be concluded that assigning students to enroll in the enhanced program for two years has a differ-ent impact on their reading achievement than assigning them to enroll in one year of the enhanced program.

Figure ES.3 places these impact estimates in the context of the actual and expected two-year achievement growth of students in the enhanced program group. It shows the two-year growth for students in the enhanced program and what their expected growth would have been had they been assigned to the regular program. It also shows the test score growth for a nation-ally representative sample of students. The test scores of students in the enhanced program group grew 25.1 points in the first year and 17.7 points in the second, for a total of 42.8 points. However, the test scores of students in the regular program group also grew, by 28.7 points in the first year and 19.7 points in the second, for a total of 48.4 points. The difference in growth rates between the two program groups produces the two-year impact estimate mentioned above, a -5.6-point difference after two years (in favor of the regular program group).

As in the math analysis, the association between receiving two years of enhanced services and reading achievement was estimated using nonexperimental methods, by statistically adjusting for the 43 percent of students in the enhanced program group who did not attend the program in the second year.¹⁶ Consistent with the experimental estimate for the impact of offering students two years of enhanced services, the association between receiving enhanced academic services for two consecutive years and SAT 10 total reading scores is negative and statistically significant (-7.5 scaled score points, p-value = 0.04). These findings suggest that two years of enhanced after-school services — whether offered or received — produces significantly fewer gains on reading achievement than two years in the regular program group.

The analysis also explored whether the one-year impact estimates for each of the 12 centers are correlated with factors related to program implementation or what the students experience during the regular school day. The analysis is based on center-level impacts in both years of the study (i.e., 24 center-level impacts) and examines whether the impact of one year of enhanced services on SAT 10 total reading scores in each after-school center is associated with (1) the characteristics of the school that housed the after-school center and (2) the characteristics of a center’s implementation of the enhanced program. Program impacts on total reading scores are not systematically correlated jointly with either the set of school context and implementation characteristics or with any of those characteristics individually. Thus, the measured local characteristics do not highlight any lessons for settings in which the program will be more effective than average.

Top