Skip Navigation
The Evaluation of Enhanced Academic Instruction in After-School Programs
NCEE 2009-4077
September 2009

Key Findings

Enhanced Math Program

  • One year of enhanced instruction produces positive and statistically significant impacts on student achievement. The impacts in the 15 centers on SAT 10 total math scores are 3.5 scaled score points in the first year (which is statistically significant) and 3.4 scaled score points in the second year of operations (which is not statistically significant). However, the difference in impacts between implementation years is not statistically significant. The impact of 3.5 scaled score points represents approximately one monthís worth of extra math learning.
  • Two years of the enhanced program produces no additional achievement benefit beyond the one-year impact. Several different analyses support this conclusion. An experimental analysis using the two-year sample finds that the estimated impact of offering students the opportunity to enroll in the enhanced program for two consecutive years (2.0 scaled score points, p-value = 0.52) and the estimated impact on these students of their first year of enrollment in the enhanced program (5.2 scaled score points, p-value = 0.07) are not statistically significantly different (p-value = 0.28). A nonexperimental analysis finds that this remains the case after adjustments are made for students in the enhanced program group who did not attend the enhanced program at all in the second year.
  • There was program fidelity across both years of implementation. Certified teachers were hired, trained, and provided paid preparation time as intended; class sizes were approximately 9 students per instructor (intended ratio was 10 students per instructor); and reports from teachers and district coordinators (i.e., locally based technical assistance staff) indicated that teachers were able to cover the expected material in a class session.
  • Students in the enhanced program received math instruction that was more structured and intensive than regular after-school program students. Students in the enhanced program group were offered formal instruction in math for three hours per week, and students in the regular program received a mix of homework help and other services not focused on math — although 17 percent of regular program group students in the first year, and 27 percent in the second, received some form of math instruction. Overall, during their first year of participation, enhanced program students received between 42 and 48 more hours of after-school math instruction than did students in the regular after-school program, which converts to a 26 to 30 percent increase in formal instruction in math over the course of the school year.
  • No clear lessons emerge for program improvement or targeting the program in particular types of schools. Analysis exploring the associations between center-level impacts and the characteristics of schools in which centers operated and the implementation of the program produced no strong associations with clear programmatic implications.

Enhanced Reading Program

  • The enhanced program has no impact on total reading test scores after one year of participation. This is true in both implementation years in these 12 centers.
  • Two years of participation produces significantly fewer gains in reading achievement for students in the enhanced program group. Experimental analysis finds that offering students two years of the enhanced reading program has a negative and statistically significant impact on their total reading scores. Nonexperimental analysis suggests that this remains the case even after statistical adjustments are made for students in the enhanced program group who did not actually attend the enhanced program in the second year.
  • Though the reading program was staffed and supported as planned, implementation issues — especially related to the pacing of lessons — occurred in both years. As with math, certified teachers were hired, trained, and provided paid preparation time as intended, and class sizes were approximately 9 students per instructor (intended ratio was 10 students per instructor). However, lesson pacing was a problem in the first year and continued to be in the second year in at least four of the districts.3
  • Students in the enhanced program received reading instruction that was more structured and intensive than regular after-school program students. Students in the enhanced program group were offered formal reading instruction for three hours per week, and most students in the regular program received a mix of homework help and other services not focused on reading — although 17 percent of regular program group students in the first year, and 12 percent in the second, received some form of reading instruction. Overall, during their first year of participation, enhanced program students received between 54 and 56 more hours of after-school instruction in reading than did students in the regular after-school program, which converts to 22 to 23 percent more formal instruction in reading over the course of the school year.
  • No systematic relationship exists between center-level impacts and program implementation or the local school context.

Top

3 In the second year, district staff who helped in implementing the model were asked if pacing continued to be a problem for staff. Of all 10 district staff interviewed, four said it was a problem, four said it was not, and two did not answer the question, so itís not clear whether it was or was not a problem in those last two districts.