Chapter 2: The Theoretical Framework Underlying the ITT Parameter

We consider two-level clustered designs where students are nested within units (such as schools, classrooms, or districts) that are randomly assigned to a single treatment or control group—the most common designs used in large-scale RCTs in the education field. The results that are presented for twolevel designs, however, can be collapsed to obtain results for nonclustered designs where students are the unit of random assignment. This is because nonclustered designs are a special case of clustered designs where every cluster has one student and there is no within-cluster variance.

We consider a "superpopulation" version of the Neyman-Rubin causal inference model (see Rubin 1974; Imbens and Rubin 2007; Schochet 2008). It is assumed that the sample contains n units (groups), with np treatment units and n(1-p) control units, where p is the sampling rate to the treatment group (0 <p<1). Let W_Ti be the "potential" unit-level outcome for unit i when assigned to the treatment condition and W_Ci be the potential outcome for unit i in the control condition. These potential outcomes are assumed to be random draws from potential treatment and control outcome distributions in the study population with means μ_T and μ_C, respectively, and common variance σ_B². It is assumed that the potential outcomes for each unit are independent of the treatment status of other units.

Suppose next that m_i students are sampled from the student superpopulation within study unit i. Let Y_Tij and Y_Cij be student-level potential outcomes (conditional on unit-level potential outcomes) that are random draws from potential outcome distributions with means WTi and WCi, respectively, and common variance σ_W² >0.

Under this causal inference model, the difference between the two potential outcomes, (W_Ti -W_Ci), is the unit-level treatment effect for unit i, and the ITT (or average treatment effect) parameter is E(W_Ti -W_Ci) =μ_T -μ_C The unit-level treatment effects, and hence, the ITT parameter, cannot be calculated directly because for each unit and student, the potential outcome is observed in either the treatment or control condition, but not in both. Formally, if T_i is a treatment status indicator variable that equals 1 for treatments and 0 for controls, then the observed outcome for a unit, w_i, can be expressed as follows:

(1) w_i = T_iW_Ti +(1-T_i) W_Ci.

Similarly, the observed outcome for a student y_ij is:

(2) y_ij = T_iY_Tij +(1-T_i) Y_Ci.

The simple equations in (1) and (2) form the basis for the estimation models that are considered in this report.

The terms in (1) can be rearranged to create the following regression model:

(3) y_ij =α₀ +α_ITTT_i +(u_i +e_ij), where

1. α₀ = μ_C and α_ITT =μ_T -μ_C (the ITT parameter) are coefficients to be estimated;

2. u_i =T_i (W_Ti -μ_T) +(1-T_i) W_Ci -μ_C) is a unit-level error term with mean zero and between-unit variance σ_B² that is uncorrelated with T_i; and

3. e_ij =T_i (Y_Tij -W_Ti) +(1-T_i) Y_Cij -W_Ci) is a student-level error term with mean zero and within-unit variance σ_W² that is uncorrelated with u_i and T_i.

Importantly, (3) can also be derived using the following two-level hierarchical linear model (HLM) (Bryk and Raudenbush 1992):

Level 1: y_ij =w_i +e_ij

Level 2: w_i =α_i +α_ITT +u_i

where Level 1 corresponds to students and Level 2 to units. Inserting the Level 2 equation into the Level 1 equation yields (3). Thus, the HLM approach is consistent with the causal inference theory presented above.

Finally, baseline covariates can be included in (3) as "irrelevant" variables to improve the precision of the impact estimates, which yields the following estimation model:

(4) y_ij =α₀ +α_ITTT_i +(X_ij -X_i′γ +Z_i′δ +(u_i* +e_ij*

where X_ij is a vector of student-level baseline covariates that is centered around the unit-level covariate mean X_i; γ is a parameter vector that is associated with X_ij;Z_i is a vector of unit-level baseline covariates (that could include X_i and stratum indicators) with associated parameter vector δ; and u_i* and e_ij* are error terms that are now conditional on the covariates. We center the X_ij covariates around the unit-level means so that we can separately identify the effects of covariates on the within- and betweenunit variance components.

Top