Skip Navigation
The Evaluation of Enhanced Academic Instruction in After-School Programs
NCEE 2009-4077
September 2009

Study Design

After-School Centers in the Study

At the start of the study, after-school centers were chosen based on their expressed interest and their ability to implement the program and research design. Assignment of centers to either the reading or the math enhanced program was based on a combination of local preferences, including knowledge of their student needs, sufficient contrast between current academic offerings in the subject area and the enhanced program, and their ability to meet the study sample needs. The 27 after-school centers that voluntarily agreed to participate in the study for a second year are located in 11 sites within 10 states and include schools and community-based organizations in a variety of municipalities (rural, urban, and suburban) across the country. They provided the same type of enhanced after-school program (math or reading) as they had provided in the first year of the study.

Student Sample and Random Assignment

The research design uses a lottery-like process (random assignment) to offer students one of two alternative types of academic support during a 45-minute block of time: the enhanced after-school academic services being tested in this project or the regular after-school services offered in their center. Regular after-school services consisted most commonly of help with homework — although, across both years of implementation, 22 percent of regular program staff in math centers reported providing some form of academic instruction in math and 14 percent of regular program staff in reading sites reported providing some form of academic instruction in reading.

The target population for the study is students in second through fifth grades who are behind grade level in reading or math but not by more than two years. The study sample was recruited from students enrolled in after-school programs and identified by local staff as in need of supplemental academic support to meet local academic standards. Those whose parents then consented to be part of the study and applied for their children to participate in the enhanced program were included in the study sample. Given that instruction in these programs is provided in a small-group format and is not specifically developed to address special needs, students with severe learning disabilities and behavior problems or who could not receive instruction in English were excluded from the sample.

This study is based on a two-stage random assignment design of students, in which students were randomly assigned by grade within each after-school center on two separate occasions — once at the beginning of the first year of the study (first stage in fall 2005, see Stage 1 of Figure ES.1) and then again at the beginning of the second study year (second stage in fall 2006, see Stage 2 of Figure ES.1). (For more details on this two-stage random assignment design, see Box ES.1.) As a result, the sample includes: students who applied to the first year of the study (as described above) and were randomly assigned to either the enhanced program group (E1) or the regular program group (R1) and are referred to throughout this report as Cohort 1; students who were not offered the enhanced program in the first year and were applicants in the second year who were either offered the enhanced program (R1E2 and NE2 applicants) or the regular program (R1R2 and NR2 applicants) and are referred to throughout this report as Cohort 2; and students who, through the two-stage random assignment design, were randomly assigned to the enhanced program in both implementation years (E1E2 group in Figure ES.1) or assigned to the regular program in both years (R1R2 group) and are referred to as the two-year sample. Cohort 1 and Cohort 2 student samples are used to estimate the one-year intent-to-treat impact of the program in the first and second implementation years, respectively. The two-year sample is used to estimate the intent-to-treat impact of offering students the enhanced program for two consecutive years.

Impact findings are based on data collected from students, regular-school-day teachers, and school records. The Stanford Achievement Test, Tenth Edition (SAT 10), abbreviated battery for math or reading (depending on the intervention implemented), was administered to students at the beginning and end of the school year to measure the gains in achievement. For second-and third-grade students in the reading sample (and all students in the second year), the Dynamic Indicators of Basic Early Literacy Skills (DIBELS) was also administered to measure fluency.

When estimating the impact of one year of exposure to the enhanced instruction separately for each implementation year, the study is equipped to detect an impact of 0.10 standard deviation in math and 0.11 standard deviation in reading in the first year of implementation, and an impact of 0.15 standard deviation in math and 0.14 standard deviation in reading in the second year of implementation.4 The study is also equipped to detect the impact of offering students two consecutive years of the program that is as small as a 0.21 standard deviation for the math program and 0.23 standard deviation for the reading program.

The following two sections present findings for the enhanced math and reading programs, respectively, based on the 27 after-school centers that participated in both years of the study.

Top

4 The number of students in the sample is a crucial factor that determines the degree to which the impacts on student achievement and other outcomes can be estimated with enough precision to reject with confidence the hypothesis that the program had no effect. In general, larger sample sizes provide more precise impact estimates. A common way to represent statistical precision is through the “minimum detectable effect size” (MDES). Formally, the MDES is the smallest true program impact (scaled as an effect size) that can be detected with a reasonable degree of power (80 percent) for a given level of statistical significance (5 percent).