Chapter 5: Multilevel RD Designs

In this section, results from above are generalized to multilevel designs where data are analyzed at the student rather than group level. Designs II and III are discussed first, followed by a discussion of Designs IV to VI.

Designs II and III

The causal inference theory discussed above can be extended to the two-level design where students are nested within units that are assigned to a research status. As before, let Y_Ti and Y_Ci be unit-level potential outcomes and Score_i be unit-level assignment scores, whose joint distributions are defined as in Chapter 4.⁸ The sample contains np treatment units and n(1-p) control units.

Suppose that m students are sampled from the student superpopulation within each study unit. Let W_Tij be the potential outcome for student j in unit i in the treatment condition and W_Cij be the corresponding potential outcome for the student in the control condition. It is assumed that W_Tij and W_Cij are random draws from student-level potential treatment and control outcome distributions (that are conditional on school-level potential outcomes) with means Y_Ti and Y_Ci, respectively, and common variance σ_θ².

In what follows, the two-level RA and RD designs are discussed using this causal inference framework.

The RA Design
Under the RA design, the observed outcome for a student, w_ij^RA, can be expressed as follows:

As before, terms in (18) can be rearranged to create the following regression model:

where:

1. α₀ and α₁ (the ATE parameter) are defined as above
2. λ =T_i^RA (Y_Ti -μ_T) + (1-T_i^RA)(Y_Ci -μ_C) is a unit-level error term with mean zero and between-unit variance σ_λ² that is uncorrelated with T_i^RA
3. θ =T_i^RA (W_Tij -Y_Ti) +(1-T_i^RA)(W_Cij -Y_Ci) is a student-level error term with mean zero and within-unit variance θ_σ² that is uncorrelated with λ_i and T_i^RA

Importantly, (19) can also be derived using the following two-level hierarchical linear (HLM) model (Bryk and Raudenbush 1992):

where Level 1 corresponds to students and Level 2 corresponds to units. Inserting the Level 2 equation into the Level 1 equation yields (19). Thus, the HLM approach is consistent with the causal inference theory presented above.

Suppose that Score_i and other unit- and student-level baseline covariates are included in (19) as covariates. In this case, the asymptotic variance of the two-level (TL) OLS estimator for the ATE parameter is as follows:

where R_{RA_X_B}² is the asymptotic regression R² value for the between-variance component and R_{RA_X_W}² is the asymptotic regression R2 value for the within-variance component. These two R² values could differ depending on the nature of the covariates.

The within-school variance term in (20) is the conventional variance expression for an impact estimate under a nonclustered design. Design effects in a clustered design arise because of the first variance term, which represents the correlation of the outcomes of students within the same units (Murray 1998; Donner and Klar 2000; Raudenbush 1997). Design effects can be large because the divisor in the between-unit term is the number of units rather than the number of students.

It is common to express the variance expression in (20) in terms of the intraclass correlation (ICC) (Cochran 1963; Kish 1965), which is defined as the between-unit variance (σ_λ²) as a proportion of the total variance of the outcome measure (σ² =σ_λ² +σ_θ²):

In this formulation, design effects from clustering are small if the mean of the outcome measure does not vary much across units (that is, if ICC is small). In this case, the approach discussed above where student-level data are averaged to the unit level will provide consistent, but inefficient impact estimates. On the other hand, if the ICC is large (that is, close to 1), then using unit averages or individual studentlevel data will produce impact estimates with similar levels of precision. Specific ICC (and R²) values will depend on the design.

The RD Design
Results from Chapter 4 for the RD design can also be extended to the two-level model. Let the observed outcome for a student, w_ij^RD, be expressed as follows:

where the term inside the brackets is a mean zero residual term. If Y_Ti and em>Y_Ci are modeled as a linear function of the assignment scores (as in [7a] and [7b]), (22) yields the following two-level RD regression model:

where τ_i is a mean zero unit-level error term with variance σ_τ², and δ_ij is a mean zero student-level error term with variance σ_δ² that is uncorrelated with τ_i. The parameter α₁ is the same ATE_K parameter as in (8) above.

As with the RA design, (23) can be obtained using a two-level HLM model:

Inserting the Level 2 equation into the Level 1 equation yields (23).

In Appendix B, it is proved that the two-level OLS estimator α̂₁ in (23) yields a consistent estimator of the ATE_K parameter assuming that the model is specified correctly. This result holds even if Score_i is correlated with τ_i. Assuming that additional baseline covariates are included in the model, the asymptotic variance of this estimator is as follows (see Appendix B):

where R_{RD_X_B}² and R_{RD_X_W}² are between- and within-unit asymptotic regression R² values, respectively.

The RD Design Effect
A key finding is that the RD design effect remains at 1/(1 -ρ_TS²) under the two-level design. This is because the variances inside the brackets in (24) for the RD design equal the variances inside the brackets in (20) or (21) for the RA design. Thus, relative to the aggregated model presented above, the use of the two-level model for Designs II and III will typically improve the precision of the impact estimates for both the RD and RA designs. However, the proportional improvement in precision is the same for each design, so that the RD design effect does not change. Similar results about the RD design effect apply also for the fuzzy RD design and for calculating MDEs.

Designs IV, V, and VI

Theoretical results from the models discussed above carry over directly to Design VI, where schools are the unit of assignment and classroom effects are treated as random. Ignoring R² terms for simplicity, the variance expression for the RD impact estimator for this design is:

where ICC₁ is the intraclass correlation at the school level, ICC₂ is the intraclass correlation at the classroom level, n is the number of schools, and c is the number of study classrooms per school. This expression is a product of the variance expression for the RA design and the RD design effect 1/(1 -ρ_TS²).

The situation is somewhat different for Design IV where the unit of assignment is at the student level and site effects are treated as random. For this design, it is assumed that treatment effects are constant within sites, but not across sites. Instead, site-level treatment effects are assumed to be drawn from a population distribution with variance σ_I².

In this case, the design effect 1/(1 -ρ_TS²) affects the student-level variance term, but not the site-level variance term. Ignoring R² terms, the variance expression for the RD impact estimate under Design IV is:

Thus, the RD design effect is smaller for Design IV than for the other designs considered above.

A similar situation occurs under Design V, where the variance expression for the RD impact estimate is:

Top