Chapter 2. The Neyman Causal Inference Model

This chapter discusses the Neyman finite-population (FP) and super-population (SP) causal inference models under two-stage clustered designs—the most common designs used in education RCTs. The focus is on continuous outcomes. The theory is then used to derive regression equations for estimating the ATE parameters.

The Neyman Finite-Population Model for Two-Stage Clustered Designs

Consider an experimental design where n schools (or classrooms) are randomly assigned to either a single treatment or control condition. The sample contains np treatment and n(1- p) control group schools where p is the sampling rate to the treatment group (0 < p <1) . It is assumed that the sample contains m_i students from school i and that there are M = Σ_n^{i =1}m_i total students in the sample. It is assumed that student outcomes are not affected by the treatment status of other students.

It is assumed for now that the n schools and M students define the population universe—the FP model considered by Neyman for non-clustered designs. Let Y_Tij be the "potential" outcome for student j in school i in the treatment condition and Y_Cij be the potential outcome for the student in the control condition. The difference between the two fixed potential outcomes,(Y_Tij -Y_Cij) , is the student-level treatment effect, and the ATE parameter, β₁, is the average treatment effect over all students:

This ATE parameter cannot be calculated directly because potential outcomes for each student cannot be observed in both the treatment and control conditions. Formally, if T_i is a treatment status indicator variable that equals 1 for treatment schools and 0 for control schools, then the observed outcome for a student, y_ij, can be expressed as follows:

(2) y_ij =T_iY_Tij +(1 -T_i)Y_Cij.

Importantly, the potential outcomes in (2) are fixed and the only source of randomness is T_i. Thus, under the Neyman model, the ATE parameter pertains only to those students and schools at the time the study was conducted. Stated differently, the impact findings have internal validity but do not necessarily generalize beyond the study sample. This approach can be justified on the grounds that schools are usually purposively selected for education RCTs, and thus, may be a self-selected sample of schools that are willing to participate, and that are deemed to be suitable for the study based on their student and teacher populations and typical service offerings. Similarly, students in the study sample may not be representative of all students in the study schools, because they could be a potentially nonrandom subset of students whose parents consented to participate in the study, who provided follow-up data, and who did not leave the study schools between baseline and followup.¹

Under this fixed population scenario, researchers are to be agnostic about whether the study results have external validity. Policymakers and other users of the study results can decide whether the impact evidence is sufficient to adopt the intervention on a broader scale, perhaps by examining the similarity of the observable characteristics of schools and students in the study samples to their own contexts, and using results from subgroup and implementation analyses.

Following the approach for non-clustered designs used by Freedman (2008) and Schochet (2009), a regression model for (2) can be constructed by re-writing (2) as follows:

(3) y_ij =β₀ +β₁(T_i -p) +η_ij, where

• β₀ =pY_T +(1 -p)Y_C and β =Y_T -Y_C are parameters to be estimated
• η_ij =α_ij + τ_ji(T_i -p) is an "error" term, where α_ij = p(Y_Tij-Y_T) +(1-p)(Y_Cij -Y_C) and τ_ij = (Y_Tij - Y_T) -(Y_Cij -Y_C).²

The error term η_ij is a function of two terms: (1) α_ij, the expected observed outcome for the student relative to the expected mean observed outcome; and (2) τ_ij, the student-level treatment effect relative to the ATE. Note that α_ij and τ_ij sum to zero over all students. This model is non-parametric because it does not depend on the distributions of the potential outcomes.

The model in (3) does not satisfy key assumptions of the usual random effects model, because η_ij does not have mean zero (over all possible treatment assignment configurations), and, to the extent that τ_ij varies across students, η_ij is heteroscedastic, Cov(η_ijη_i′j′) is not constant for students in the same schools, Cov(η_ijη_ij) is nonzero for students in different schools (for i≠′i, j≠′j), and η_ij is correlated with the i-regressor (T_i -p):

E(η_ij) =α_ij, Var(η_ij) =τ_ij²p(1 -p), Cov(η_ijη_ij′) =τ_ijτ_ij′p(1 -p),
Cov(η_ijη_i′j′) =τ_ijτ_i′j′p(1 -p)/(n -1), E[(T_i -p_i)η_ij′] =τ_ijp(1 -p).

Note that in this model, the error terms for students within the same schools are correlated only because they have the same treatment status, not because they face similar environments.

Importantly, the model in (3) should not be confused with a fixed effects model, where cluster effects are treated as fixed, and cluster-level dummy variables are included in the model. Rather, the model treats cluster-level effects as random due to the randomness of treatment status in the model error term.

Finally, (3) implicitly assumes that schools are weighted by their student sample sizes. An alternative specification is to weight schools equally. In this case, the ATE parameter is β₁ =Y_T -Y_C, where Y_T= (1/n) Σ_i=1ⁿ Σ_j=1^m_i and Y_C(1 /n)Σ_i=1ⁿ Σ_j=1^m_i(Y_Cij /m_i are averages of school-level means. This ATE parameter pertains to the average school effect in the sample rather than to the average student effect. This weighting scheme will result in different impact estimates than the unweighted analysis if student sample sizes vary across schools and impacts vary by school sample size.

The Super-Population Model for Two-Stage Clustered Designs

We now consider a SP version of the Neyman causal inference model where the study schools and students are assumed to be random samples from broader populations (see Imbens and Rubin 2007 and Schochet 2008, 2009). This framework is typically used to estimate impacts under clustered RCTs in the education area, and is consistent with popular linear mixed model approaches, such as HLM.

Under this framework, students are nested within schools. Let Z_Ti be the potential outcome (mean posttest score) for school i in the treatment condition and Z_Ci be the potential outcome for school i in the control condition. Potential outcomes for the n study schools are assumed to be random draws from potential treatment and control outcome distributions in the study super-population. It is assumed that means and variances of these distributions are finite and denoted by μ_τ and σ_μτ² for potential treatment outcomes and μ_C and σ_uC²for potential control outcomes. These two outcome distributions also define the distribution of school-level treatment effects in the super-population, which are assumed to have mean μ_τ² and variance σ_τ².

Suppose next that m_i students are sampled from the student super-population in study school i. The potential student-level outcomes Y_Tij and Y_Cij are now assumed to be random draws from student-level potential outcome distributions (which are conditional on school-level potential outcomes) with respective means Z_Ti and Z_Ci and respective variances σ_eT² >0 and σ_eC².

Under the SP model, the ATE parameter is μ_τ =E(Z_Ti -Z_Ci) = μ_T -μ_C.Thus, the impact findings are now assumed to generalize to the super-population of schools that are "similar" to the study schools. How should one interpret this super-population? Does it pertain to the study schools over the "long term" for a broader universe of students and school staff that change over time? Does it pertain to a broader set of schools in the study districts? To similar schools nationwide? The answers to these questions will likely depend on the context (and may not exist), but researchers should be aware that the usual approach for estimating treatment effects in education research makes the implicit assumption of external validity to a school universe that is likely to be vaguely defined. Nonetheless, this approach can be justified on the grounds that policymakers may generalize the findings anyway, especially if the study provides a primary basis for deciding whether to implement the tested interventions more broadly. Furthermore, this approach is more consistent with the Bayesian view that assessing intervention effects is a dynamic process that takes place in a context of continuously increasing knowledge.

As before, we can use (2) to express observed student outcomes in terms of potential outcomes, and can rearrange terms to yield the following regression model:

(4) y_ij =α₀ +α₁T_i +(u_i +e_ij), where
α₀ =μ_C and α₁ =μ_T -μ_C (the ATE parameter) are coefficients to be estimated
u_i =T_i(Z_Ti -μ_T) +(1 -T_i)(Z_Ci -μ_C)is a school-level error term where E(u_i) =0,
E(T_iu_i) =0, Var(u_i|T_i =1) =σ_uT², and Var(u_i|T_i =0) =σ_uC²
e_ij =T_i(Y_Tij -Z_Ti) +(1 -T_i)(Y_cij -Z_Ci) is a student-level error team where E(e_ij) =0,
E(T_ie_ij) =E(u_ie_ij) =0, Var(e_ij|T_i =1) =σ_eC².

Furthermore, if we define δ_ij =u_i +e_ij as the total error term:

Var(δ_ij|T_i =1) =σ_uT² +σ_eT², Var(δ_ij|T_i =0) =σ_uC²,σ_eC² , Cov(δ_ij,δ_i′j′) =0, Cov(δ_ij′|T_i =1),σ_uT², Cov(δ_ij′|T_i =0),σ_uC²

Thus, this model is the usual random effects model with an exchangeable block diagonal variance-covariance matrix for the error vector except that variances and covariances are allowed to differ for treatments and controls.

Finally, note that (4) can also be derived using the following two-level HLM model (Bryk and Raudenbush, 1992):

Level 1:y_ij =z_i +e_ij
Level 2: z_i =α₀ +α₁T_i +u_i

where z_i =T_iZ_Ti +(1-T_i)Z_Ci is the observed school-level outcome, Level 1 corresponds to students, and Level 2 to units. Inserting the Level 2 equation into the Level 1 equation yields (4). Thus, the HLM approach is consistent with the SP causal inference theory.

Top