Highlights Archive for September 2009
A distinguishing feature of after-school programs supported by 21st CCLC funds has been the inclusion of an academic component. Yet, findings from the National Evaluation of the 21st CCLC program indicate that, on average, the early 21st CCLC program grants had limited effects on students’ academic outcomes (Dynarski and others, 2003; Dynarski and others, 2004; James-Burdumy et al., 2005). In order to provide information about potential academic programming in the afterschool setting, this study conducted by MDRC includes an evaluation of Adventure Island (a reading program developed by the Success for All Foundation) and Mathletics (a math program developed by Harcourt School Publishers), two structured academic programs specifically adapted for the after-school setting. Students attending an afterschool program were assigned by lottery to either receive the structured academic programming (the enhanced program) or the after-school programming regularly offered. This second and final report includes first and second year information from 27 centers (12 providing the reading program and 15 providing the math program).
The report contains the following key findings:
- One year of enhanced math instruction produced positive impacts on student achievement of approximately one month of extra learning; a second year produced no additional achievement benefit. The impacts in the 15 centers are 3.5 scaled score points in the first year for Cohort 1 (which is statistically significant) and 3.4 scaled score points for in the second year of operations for Cohort 2 (which is not statistically significant, p-value =0.07). The difference in impacts between implementation years does not statistically differ.
- The enhanced program had no impact on total reading test scores after one year of participation; a second year of the program produced fewer gains in reading achievement for students in the enhanced program group. There were no positive one-year impacts in both implementation years in these 12 centers. Experimental analysis finds that offering students two years of the enhanced reading program has a negative and statistically significant impact on their total reading scores. Nonexperimental analysis suggests that this remains the case even after statistical adjustments are made for the fact that 43 percent of students in the enhanced program group did not actually attend the enhanced program in the second year.
- Both programs were staffed and supported as intended; the math program was generally implemented with fidelity; the reading program experienced some implementation issues across both years. For both programs, certified teachers were hired, trained, and provided paid preparation time as intended; class sizes approximated the intended ratio of 10 students per instructor; and reports from teachers and locally-based technical assistance staff in the math sites indicated that teachers were able to cover the expected material in a class session; reports from the reading sites indicated that lesson pacing was a problem in the first year and continued to be a problem in the second year, at least in four of the 10 participating districts.
- Students in the enhanced program in both the reading and math sites received instruction that was more structured and intensive than regular after-school program students. Students in the enhanced program group were offered formal instruction for 3 hours per week, and most students in the regular program received a mix of homework help and other services not focused on the topic area offered in the enhanced instruction. Within a single school year, the enhanced math students received the equivalent of a 26 to 30 percent increase in formal instruction in math over the course of the school year. The enhanced reading students received the equivalent of 22 to 23 percent more formal instruction in reading over the course of the school year.
