After-School Centers in the Study
At the start of the study, after-school centers were chosen based on their expressed
interest and their ability to implement the program and research design. Assignment
of centers to either the reading or the math enhanced program was based on a combination
of local preferences, including knowledge of their student needs, sufficient contrast
between current academic offerings in the subject area and the enhanced program,
and their ability to meet the study sample needs. The 27 after-school centers that
voluntarily agreed to participate in the study for a second year are located in
11 sites within 10 states and include schools and community-based organizations
in a variety of municipalities (rural, urban, and suburban) across the country.
They provided the same type of enhanced after-school program (math or reading) as
they had provided in the first year of the study.
Student Sample and Random Assignment
The research design uses a lottery-like process (random assignment) to offer students
one of two alternative types of academic support during a 45-minute block of time:
the enhanced after-school academic services being tested in this project or the
regular after-school services offered in their center. Regular after-school services
consisted most commonly of help with homework — although, across both years
of implementation, 22 percent of regular program staff in math centers reported
providing some form of academic instruction in math and 14 percent of regular program
staff in reading sites reported providing some form of academic instruction in reading.
The target population for the study is students in second through fifth grades who are behind grade level in reading or math but not by more than two years. The study sample was recruited from students enrolled in after-school programs and identified by local staff as in need of supplemental academic support to meet local academic standards. Those whose parents then consented to be part of the study and applied for their children to participate in the enhanced program were included in the study sample. Given that instruction in these programs is provided in a small-group format and is not specifically developed to address special needs, students with severe learning disabilities and behavior problems or who could not receive instruction in English were excluded from the sample.
This study is based on a two-stage random assignment design of students, in which students were randomly assigned by grade within each after-school center on two separate occasions — once at the beginning of the first year of the study (first stage in fall 2005, see Stage 1 of Figure ES.1) and then again at the beginning of the second study year (second stage in fall 2006, see Stage 2 of Figure ES.1). (For more details on this two-stage random assignment design, see Box ES.1.) As a result, the sample includes: students who applied to the first year of the study (as described above) and were randomly assigned to either the enhanced program group (E1) or the regular program group (R1) and are referred to throughout this report as Cohort 1; students who were not offered the enhanced program in the first year and were applicants in the second year who were either offered the enhanced program (R1E2 and NE2 applicants) or the regular program (R1R2 and NR2 applicants) and are referred to throughout this report as Cohort 2; and students who, through the two-stage random assignment design, were randomly assigned to the enhanced program in both implementation years (E1E2 group in Figure ES.1) or assigned to the regular program in both years (R1R2 group) and are referred to as the two-year sample. Cohort 1 and Cohort 2 student samples are used to estimate the one-year intent-to-treat impact of the program in the first and second implementation years, respectively. The two-year sample is used to estimate the intent-to-treat impact of offering students the enhanced program for two consecutive years.
Impact findings are based on data collected from students, regular-school-day teachers, and school records. The Stanford Achievement Test, Tenth Edition (SAT 10), abbreviated battery for math or reading (depending on the intervention implemented), was administered to students at the beginning and end of the school year to measure the gains in achievement. For second-and third-grade students in the reading sample (and all students in the second year), the Dynamic Indicators of Basic Early Literacy Skills (DIBELS) was also administered to measure fluency.
When estimating the impact of one year of exposure to the enhanced instruction separately for each implementation year, the study is equipped to detect an impact of 0.10 standard deviation in math and 0.11 standard deviation in reading in the first year of implementation, and an impact of 0.15 standard deviation in math and 0.14 standard deviation in reading in the second year of implementation.4 The study is also equipped to detect the impact of offering students two consecutive years of the program that is as small as a 0.21 standard deviation for the math program and 0.23 standard deviation for the reading program.
The following two sections present findings for the enhanced math and reading programs, respectively, based on the 27 after-school centers that participated in both years of the study.