Chapter 3: Considered Designs
The analysis presented below applies to commonly-used clustered designs in the education
field where one of the following "units" is assigned to a single treatment or control
group: school districts, schools, classrooms or students. In these designs, students
are nested within higher-level units (groups).
Clustering in multilevel designs comes from two potential sources: (1) the assignment
of units to a research condition, and (2) the random sampling of units
from a broader universe of units before or after treatment assignments take place.
This paper considers the following designs that combine these two sources of clustering
(that are ordered based on design structure):
- Students are the unit of assignment and site (school or district)
effects are fixed. In some designs, students in purposively-selected schools
or districts are randomly assigned directly to a research group. For example, in
the Impact Evaluation of Charter School Strategies (Gleason
and Olsen 2004), within each charter school area that volunteered for the
study, students interested in attending a charter school were randomly assigned
through a lottery to either a treatment group (who were allowed to enroll in a charter
school) or a control group (who were not). Under these designs, sites can be treated
as fixed strata if the impact results are to be viewed as pertaining to the study
sites only. To estimate these models, impacts can be estimated using a pooled model
where the covariates include treatment status and site indicators (and perhaps,
site-by-treatment interactions); the error structure would include random student-level
- Classrooms are the unit of assignment and school effects are fixed.
Classroom-based designs are appropriate for interventions that are administered
at the classroom level and where potential spillover effects of the intervention
from treatment to control group classrooms are deemed to be small. If schools are
purposively selected for the study, school effects could be treated as fixed strata.
This design was used in the Evaluation of the Effectiveness of Educational Technology
Interventions (Dynarski and Agodini 2003) where
teachers in participating volunteer schools were randomly assigned to use a technology
or not. In estimating these models, random classroom effects would be included in
the model error structure, and school indicators (and perhaps, school-by-treatment
interactions) would be included as model covariates.
- Schools are the unit of assignment and no random classroom effects.
School-based designs are common in the education field, and are often preferred
over classroom-based designs because of concerns over potential spillover effects.
These designs are also necessary for testing interventions that can affect the entire
school (such as those that aim to change the school climate). The exclusion of random
classroom effects can be justified if students are sampled from all targeted classrooms
within the study schools. To estimate these models, random school effects would
be included in the error structure in the HLM models. A variant of this design is
if school districts are the unit of assignment and students are selected within
districts without regard to their schools or classrooms.
- Students are the unit of assignment and site effects are random.
This design is similar to Design I, except that sites are considered to be randomly
sampled from a broader universe of sites, so that study results are to be viewed
as generalizing outside the site sample (that is, as being externally valid). For
estimation, random site and site-by-treatment interaction terms would be included
in the error structure in the HLM models.
- Classrooms are the unit of assignment and school effects are random.
This design is a modification to Design II where school effects are treated as random.
For estimation, the model error structure would include random classroom, school,
and school-by-treatment interaction terms.
- Schools are the unit of assignment and classroom effects are random.
This design is appropriate if classrooms within study schools are sampled for the
study, or if all classrooms are included in the study but are considered to be sampled
from a larger classroom population. For estimation, the model error structure would
include random school and classroom effects.
These designs are discussed in more detail in Schochet (2008)
in the context of RA designs using a unified HLM framework.
To simplify the presentation and fix concepts, I first discuss the theory underlying
the RD design for Design I and Designs II and III where the analysis is conducted
using data that are averaged to the unit of treatment assignment (classrooms,
schools, or districts). These designs are referred to as "aggregated" designs. I
then discuss "multilevel designs" that include Designs II and III where the analysis
is conducted using student-level data and Designs IV to VI.
In what follows, the RD design is discussed in the context of the causal inference
theory underlying RA designs (Neyman 1923,
Rubin 1974, Holland 1986,
Imbens and Rubin 2007, Schochet 2007).
This framework is then used to discuss impact and variance estimation methods that
are required to calculate MDEs.