Chapter 3: Considered Designs

The analysis presented below applies to commonly-used clustered designs in the education field where one of the following "units" is assigned to a single treatment or control group: school districts, schools, classrooms or students. In these designs, students are nested within higher-level units (groups).

Clustering in multilevel designs comes from two potential sources: (1) the assignment of units to a research condition, and (2) the random sampling of units from a broader universe of units before or after treatment assignments take place. This paper considers the following designs that combine these two sources of clustering (that are ordered based on design structure):

1. Students are the unit of assignment and site (school or district) effects are fixed. In some designs, students in purposively-selected schools or districts are randomly assigned directly to a research group. For example, in the Impact Evaluation of Charter School Strategies (Gleason and Olsen 2004), within each charter school area that volunteered for the study, students interested in attending a charter school were randomly assigned through a lottery to either a treatment group (who were allowed to enroll in a charter school) or a control group (who were not). Under these designs, sites can be treated as fixed strata if the impact results are to be viewed as pertaining to the study sites only. To estimate these models, impacts can be estimated using a pooled model where the covariates include treatment status and site indicators (and perhaps, site-by-treatment interactions); the error structure would include random student-level terms only.
2. Classrooms are the unit of assignment and school effects are fixed. Classroom-based designs are appropriate for interventions that are administered at the classroom level and where potential spillover effects of the intervention from treatment to control group classrooms are deemed to be small. If schools are purposively selected for the study, school effects could be treated as fixed strata. This design was used in the Evaluation of the Effectiveness of Educational Technology Interventions (Dynarski and Agodini 2003) where teachers in participating volunteer schools were randomly assigned to use a technology or not. In estimating these models, random classroom effects would be included in the model error structure, and school indicators (and perhaps, school-by-treatment interactions) would be included as model covariates.
3. Schools are the unit of assignment and no random classroom effects. School-based designs are common in the education field, and are often preferred over classroom-based designs because of concerns over potential spillover effects. These designs are also necessary for testing interventions that can affect the entire school (such as those that aim to change the school climate). The exclusion of random classroom effects can be justified if students are sampled from all targeted classrooms within the study schools. To estimate these models, random school effects would be included in the error structure in the HLM models. A variant of this design is if school districts are the unit of assignment and students are selected within districts without regard to their schools or classrooms.
4. Students are the unit of assignment and site effects are random. This design is similar to Design I, except that sites are considered to be randomly sampled from a broader universe of sites, so that study results are to be viewed as generalizing outside the site sample (that is, as being externally valid). For estimation, random site and site-by-treatment interaction terms would be included in the error structure in the HLM models.
5. Classrooms are the unit of assignment and school effects are random. This design is a modification to Design II where school effects are treated as random. For estimation, the model error structure would include random classroom, school, and school-by-treatment interaction terms.
6. Schools are the unit of assignment and classroom effects are random. This design is appropriate if classrooms within study schools are sampled for the study, or if all classrooms are included in the study but are considered to be sampled from a larger classroom population. For estimation, the model error structure would include random school and classroom effects.

These designs are discussed in more detail in Schochet (2008) in the context of RA designs using a unified HLM framework.

To simplify the presentation and fix concepts, I first discuss the theory underlying the RD design for Design I and Designs II and III where the analysis is conducted using data that are averaged to the unit of treatment assignment (classrooms, schools, or districts). These designs are referred to as "aggregated" designs. I then discuss "multilevel designs" that include Designs II and III where the analysis is conducted using student-level data and Designs IV to VI.

In what follows, the RD design is discussed in the context of the causal inference theory underlying RA designs (Neyman 1923, Rubin 1974, Holland 1986, Imbens and Rubin 2007, Schochet 2007). This framework is then used to discuss impact and variance estimation methods that are required to calculate MDEs.

Top