Skip Navigation
The Evaluation of Enhanced Academic Instruction in After-School Programs

NCEE 2008-4021
June 2008

Early Findings for Reading

Adventure Island, the reading model put in place in 25 after-school centers, had the following first-year findings:

  • The enhanced reading program was implemented as intended (in terms of staff characteristics, training, and usage of instructional materials).
  • Students received an average of 176 minutes of reading instruction per week in the reading centers.
  • Reading instructors reported that it was difficult to include all aspects of the reading program and maintain the intended pace of the daily lesson plan.
  • The enhanced program provided students with 20 percent more hours of reading instruction over the school year, compared with students in the regular after-school program group.
  • The students in the enhanced reading program did not experience a statistically significant impact on their performance on the SAT 10 reading test; there are positive and statistically significant program impacts on one of the two measures in the DIBELS fluency test.
  • The reading program did not produce statistically significant impacts (either positive or negative) on any of the three school-day academic behavior measures: student engagement, behavior, or homework completion.

Implementation of the Enhanced Reading Program

Overall, the strategies supporting the reading intervention were implemented as intended. Specifically, centers hired certified teachers (across the 25 centers, 99 percent of staff were certified) and operated the programs with the intended small groups of students — on average, 9 students per instructor. Instructors were trained by SFA at the beginning of the year and were provided ongoing assistance and paid preparation time.9 The district coordinator reports from classroom observations of implementation indicate that 19 percent of Alphie’s Lagoon and Captain’s Cove classes included four or fewer of the six elements identified as key to intended implementation by the developer; 13 percent of Discovery Bay and Treasure Harbor classes included three or fewer of the five core elements. Observations that included fewer than 70 percent of the core elements indicate that teachers had difficulty delivering specific aspects of the program — in particular, the methods to improve fluency and the ability to cover all the intended lesson elements in the allotted time. In addition, enhanced after-school program staff indicated that the expected pacing of instruction was problematic for daily lessons.

The Service Contrast in Reading

Students in the enhanced reading program were offered and attended a different set of services during the after-school academic time slot than students in the regular after-school program.

The enhanced program offered its students academic instruction in reading, whereas 12 percent of those in the regular after-school program group were offered academic instruction in reading, and the other students received primarily homework help and/or tutoring on multiple subjects. Ninety-nine percent of staff members providing the instruction to the enhanced group students were certified teachers, compared with 60 percent of the regular after-school program staff. Additionally, 97 percent of enhanced staff received high-quality training to carry out their work, and 95 percent received ongoing support, compared with 58 percent and 55 percent of the regular program staff, respectively. These differences are statistically significant at the 0.05 level.

Students in the enhanced program group attended, on average, 48 more hours of academic instruction over the course of the school year than the regular program group received (55.0 hours compared with 6.54 hours). Given estimates of average school-day instruction, this statistically significant finding represents an estimated 20 percent more hours of reading instruction for students in the enhanced program. Students in the enhanced program group attended 10 percent more days than those in the regular after-school program group, and this difference is statistically significant (effect size = 0.19).

Impacts of the Enhanced Reading Program

Overall, the students in the first year of the enhanced reading program did not experience a statistically significant impact on their performance level on SAT 10 reading tests (total and subtests), above and beyond the level that they would have achieved had there been no enhanced program. This is true for both the full analysis sample and the subgroups defined by grade level and prior achievement. Figure ES.2 illustrates the amount of growth for both the enhanced and the regular program groups in reading over the school year and the lack of a statistically significant difference.

On the other hand, analysis shows that the enhanced reading program produced statistically positive gains in one of two measures of fluency for the younger students in the study sample. The enhanced program group scored 3.7 points higher (effect size = 0.12) in the nonsense word fluency subtest of DIBELS, which targets the alphabetic principle. However, after accounting for multiple comparisons, the estimate is no longer statistically significant.10 The estimated impact for the oral fluency measure is not statistically significant but is positive (effect size = 0.07).

The analysis also looks at impacts on three measures of student academic behavior — How often do they not complete homework? How often are they attentive in class? How often are they disruptive in class? — for all students in the sample as well as for the two sets of subgroups. Enrollment in the enhanced program did not produce statistically significant impacts on any of these measures for either the full analysis sample or the various subgroups.

Linking Local School Context to Reading Impacts

While there was no overall statistically significant impact on academic achievement for all students in the analysis sample in the first year of the enhanced reading program, not all 25 centers in the study sample experienced this exact impact. Though the study was not designed with the power to detect impacts of Adventure Island at the level of individual centers, 11 of the 25 centers did have positive point estimates; 14 of the 25 had negative point estimates. Thus, the analysis explored the possibility of variation in impacts for students who attended different types of schools and experienced different program implementation.

Because the effectiveness of after-school instruction may be related to factors associated with program implementation or what the students experience during the regular school day, a correlational analysis was conducted to shed light on the possible moderating effects of school characteristics and factors of program implementation. Note that this analysis is nonexperimental and, thus, not causal; inferences drawn from it need to be interpreted with caution. The school characteristics included in the analysis are the hours of in-school instruction in the relevant subject, whether the school met its AYP goals, the proportion of students receiving free or reduced-price lunch, and the in-school student-to-teacher ratio. The analysis also links impacts to two factors of program implementation: the number of days over the course of the school year that the enhanced reading program was offered and whether a teacher from the enhanced program left during the school year.11 No evidence was found linking the program impact on total reading scores to any of these school environment or implementation characteristics. Additionally, the full set of characteristics is not correlated with the program impacts on the total reading SAT 10 score (p-value = 0.71).

Top


9 Enhanced reading program staff received two full days of upfront training on how to use the reading materials, including feedback from the developers in practice sessions using the materials. Ongoing support given to the enhanced program staff consisted of multiple on-site technical assistance visits (an average of three), continued support by locally based staff, and daily paid preparation time of 30 minutes.
10 The DIBELS nonsense word fluency subtest is one of six reading measures estimated for second- and third-grade students. When accounting for multiple test corrections using the Benjamini-Hochberg procedure, this estimate is no longer statistically significant. See Y. Benjamini and Y. Hochberg, “Controlling the False Discovery Rate: A New and Powerful Approach to Multiple Testing,” Journal of the Royal Statistical Society, Series B(57): 1289-1300 (1995).
11 The types of reading curricula in use in schools during the regular school day were not available in a form that allowed the grouping of centers into categories.