One year of enhanced instruction produces positive and statistically significant
impacts on student achievement. The impacts in the 15 centers on SAT 10
total math scores are 3.5 scaled score points in the first year (which is statistically
significant) and 3.4 scaled score points in the second year of operations (which
is not statistically significant). However, the difference in impacts between implementation
years is not statistically significant. The impact of 3.5 scaled score points represents
approximately one month’s worth of extra math learning.
Two years of the enhanced program produces no additional achievement benefit
beyond the one-year impact. Several different analyses support this conclusion.
An experimental analysis using the two-year sample finds that the estimated impact
of offering students the opportunity to enroll in the enhanced program for two consecutive
years (2.0 scaled score points, p-value = 0.52) and the estimated impact on these
students of their first year of enrollment in the enhanced program (5.2 scaled score
points, p-value = 0.07) are not statistically significantly different (p-value =
0.28). A nonexperimental analysis finds that this remains the case after adjustments
are made for students in the enhanced program group who did not attend the enhanced
program at all in the second year.
There was program fidelity across both years of implementation.
Certified teachers were hired, trained, and provided paid preparation time as intended;
class sizes were approximately 9 students per instructor (intended ratio was 10
students per instructor); and reports from teachers and district coordinators (i.e.,
locally based technical assistance staff) indicated that teachers were able to cover
the expected material in a class session.
Students in the enhanced program received math instruction that was more
structured and intensive than regular after-school program students. Students
in the enhanced program group were offered formal instruction in math for three
hours per week, and students in the regular program received a mix of homework help
and other services not focused on math — although 17 percent of regular program
group students in the first year, and 27 percent in the second, received some form
of math instruction. Overall, during their first year of participation, enhanced
program students received between 42 and 48 more hours of after-school math instruction
than did students in the regular after-school program, which converts to a 26 to
30 percent increase in formal instruction in math over the course of the school
year.
No clear lessons emerge for program improvement or targeting the program
in particular types of schools. Analysis exploring the associations between
center-level impacts and the characteristics of schools in which centers operated
and the implementation of the program produced no strong associations with clear
programmatic implications.
Enhanced Reading Program
The enhanced program has no impact on total reading test scores after one
year of participation. This is true in both implementation years in these
12 centers.
Two years of participation produces significantly fewer gains in reading
achievement for students in the enhanced program group. Experimental analysis
finds that offering students two years of the enhanced reading program has a negative
and statistically significant impact on their total reading scores. Nonexperimental
analysis suggests that this remains the case even after statistical adjustments
are made for students in the enhanced program group who did not actually attend
the enhanced program in the second year.
Though the reading program was staffed and supported as planned, implementation
issues — especially related to the pacing of lessons — occurred in both years.
As with math, certified teachers were hired, trained, and provided paid preparation
time as intended, and class sizes were approximately 9 students per instructor (intended
ratio was 10 students per instructor). However, lesson pacing was a problem in the
first year and continued to be in the second year in at least four of the districts.3
Students in the enhanced program received reading instruction that was more
structured and intensive than regular after-school program students. Students
in the enhanced program group were offered formal reading instruction for three
hours per week, and most students in the regular program received a mix of homework
help and other services not focused on reading — although 17 percent of regular
program group students in the first year, and 12 percent in the second, received
some form of reading instruction. Overall, during their first year of participation,
enhanced program students received between 54 and 56 more hours of after-school
instruction in reading than did students in the regular after-school program, which
converts to 22 to 23 percent more formal instruction in reading over the course
of the school year.
No systematic relationship exists between center-level impacts and program
implementation or the local school context.
3 In the second year, district staff who
helped in implementing the model were asked if pacing continued to be a problem
for staff. Of all 10 district staff interviewed, four said it was a problem, four
said it was not, and two did not answer the question, so it’s not clear whether
it was or was not a problem in those last two districts.