Study Design
This study employs a student-level random assignment design.4 By randomly assigning students, by grade, within each after-school center to either the enhanced program group to receive 45 minutes of the formal academic instruction or the regular program group to receive the regular after-school services for those 45 minutes, researchers are able to eliminate systematic differences between the two groups of students. Though chance variation may still exist, differences between the groups on the outcomes measured can be attributed to the effect of the enhanced program.
This report presents findings for the first of two years of program operations (school year 2005-2006) on the two parallel studies (one of reading and one of math). The enhanced instruction was implemented in 50 after-school centers — 25 to test the reading program and 25 to test the math program. After-school centers were chosen based on their expressed interest and their ability to implement the program and research design. Assignment of centers to either the reading or the math enhanced program was based on a combination of local preferences, including knowledge of their student needs, sufficient contrast between current academic offerings in the subject area and the enhanced program, and their ability to meet the study sample needs. The centers had to affirm that they were not already providing academic support that involved a structured curriculum or that included diagnostic assessments of children to guide instruction in the subject that they would be implementing (that is, math or reading). The after-school centers are located in 16 sites within 13 states and include schools and community-based organizations in rural areas, in towns, and within the urban fringe of or in midsize to large cities across the country. Participating centers draw students from schools with an average of 78 percent of students receiving free or reduced-price lunches (a measure of low-income status).
The target population for the study is students in second through fifth grades who are behind grade level but not by more than two years. The study sample was recruited from students enrolled in after-school programs who were identified by local staff as in need of supplemental academic support to meet local academic standards. Given that instruction in these programs is provided in a small-group format and is not specifically developed for special needs, students with severe learning disabilities or behavioral problems were excluded from the sample selected. The sample students also had to be able to receive instruction in English. Students who applied to participate in the study were randomly assigned, by grade within their center, to receive either the enhanced model of academic instruction or the services of the regular after-school program. The analysis sample for math includes 1,961 students, and the sample for reading comprises 1,828 students.
Impact findings from the first year are based on data collected from students, regular-school- day teachers, and school records. The Stanford Achievement Test, Tenth Edition (SAT 10), abbreviated battery for math or reading (depending on the intervention implemented), was administered to students at the beginning and end of the school year to measure the gains in achievement. For second- and third-grade students in the reading sample, the Dynamic Indicators of Basic Early Literacy Skills (DIBELS) was also administered to measure fluency. A survey of regular-school-day teachers was used to measure student academic behavior.
To help interpret the impact findings, this study also examines how well the special academic services were implemented and whether the enhanced program actually produced a service contrast with what the control group received in the regular after-school program. Thus, the study answers these two questions:
- Implementation. How are the after-school academic interventions implemented in the study centers?
- Service contrast. What are the measurable differences between services received by students in the enhanced program group and services received by students in the regular after-school program (or control) group?
In addition, the enhanced program was offered in a variety of types of schools. Because the effectiveness of after-school instruction may be related to what the students experience during the regular school day, a third issue is also examined:
- Linking local school context to impacts. Are factors related to local schoolday context associated with program impacts?
4 Random assignment was conducted at the student, rather than a higher, level because random assignment at the student level provided more power with which to detect impacts for a given number of after-school programs. Additionally, implementation did not need to be the whole after-school center in order for it to operate. The study team was not concerned about control group contamination because use of the programs required specific training and materials not available to the control group teachers. Furthermore, treatment and control conditions were monitored throughout the study for potential cross-contamination.