Skip Navigation
The Evaluation of Enhanced Academic Instruction in After-School Programs
NCEE 2009-4077
September 2009

Box ES.1.  Two-Stage Random Assignment Design


The study is based on a two-stage random assignment design. At the beginning of the first study year (1st stage in fall 2005, see Stage 1 of Figure ES.1), identified low-performing students who applied to the study were randomly assigned by grade within each after-school center to either the enhanced program group (E1) or the regular program group (R1), and are referred to as Cohort 1.

At the end of the first study year, IES decided to extend the study for a second study year to assess both: (1) the one-year impact of the enhanced program and whether that impact changes over time once the site and staff have experience with the program (i.e., a compari-son of the one-year impact of the program between the first and second study year), and (2) the impact of extended exposure to the enhanced program (i.e., an estimate of the two-year cumulative effect of being offered the enhanced program both years compared to being offered the regular program both years).

In order to address both these goals for the second study year, a second round of random assignment was conducted consisting of two groups of students, applicants and nonapplicants (2stage in fall 2006, see Stage 2 of Figure ES.1). The application process in the sec-ond year of the study was conducted the same as in the first year of the study. Applicants in the second year consisted of newly identified low-performing students who were new appli-cants in year 2 and students from Cohort 1 who voluntarily applied to the second year of the study. Both of these groups of student applicants in Year 2 were randomly assigned by grade within each after-school center to either the enhanced program group or the regular program group; applicants from Cohort 1 were also randomly assigned by their first year treatment status (whether they were part of the enhanced or regular after-school program group). Ran-domly assigning for a second time students who participated in the first year, rather than allowing them to maintain their initial randomly assigned grouping, ensured that those who were offered the enhanced program the first year did not receive special treatment once the study was extended.

Nonapplicants are the remaining Cohort 1 students who had participated in the first year of the study, but did not apply to the second year of the study. They too were randomly as-signed (separately from applicants) by grade and their first year treatment status within each after-school center. Randomly assigning both the applicants and nonapplicants from Cohort 1 maintains an intent-to-treat sample of Cohort 1 students who are cumulatively offered two years of the program or never offered the program. (Note, fifth-graders from Cohort 1 were excluded from the second stage of the random assignment in fall 2006 because, as sixth-graders, they were no longer eligible for the program and thus did not reapply.)