Chapter 5: Theoretical Framework

This chapter discusses the statistical theory underlying the variance-bias tradeoff associated with including pretests in the posttest impact models for two-level clustered RCTs. The theory is discussed in the context of the causal inference theory underlying RCTs (Neyman 1923; Rubin 1974; Holland 1986; Imbens and Rubin 2007; Freedman 2008; Schochet 2007).

It is assumed that students are nested within n units (schools or classrooms) that are randomly assigned to a single treatment or control group. The sample is assumed to contain np treatment units and n(1-p) control units, where p is the sampling rate to the treatment group (0<p<1).

This paper considers a "superpopulation" version of the Neyman-Rubin causal inference model (see Imbens and Rubin 2007; Schochet 2007; and Yang and Tsiatis 2001). Let Z_1Ti be the "potential" unit-level continuous posttest score for unit i in the treatment condition and Z_1Ci be the potential posttest score for unit i in the control condition. Potential posttest scores for the n study units are assumed to be random draws from potential treatment and control posttest distributions in the study population, with means μ_1T and μ_1C, respectively; a common variance σ₁² > 0 is assumed for each research group to ensure that variance estimates based on standard ordinary least squares (OLS) methods are justified by the Neyman- Rubin causal model (Freedman 2008; Schochet 2007). It is assumed that treatment assignments are independent of potential outcomes (due to random assignment), and that potential outcomes for each unit are unrelated to the treatment status of other units. Finally, let Z_0Ti, Z_0Ci, μ_0T, μ_0C, and σ₀² denote corresponding quantities for fall pretest scores, and let σ₀₁ denote the covariance between the potential pretest and posttest scores for both the treatment and control groups (which could depend on how late the pretests are collected).¹

Suppose next that m students are sampled from the student superpopulation within each study unit. Let Y_1Tij be the potential posttest score for student j in unit i in the treatment condition and Y_1Cij be the corresponding potential posttest score for the student in the control condition. Y_1Tij and Y_1Cij are assumed to be random draws from student-level potential treatment and control posttest distributions (which are conditional on school-level potential outcomes) with means Z_1Ti and Z_1Ci, respectively, and common variance τ₁² > 0. Corresponding variables for student-level pretest scores are denoted by replacing subscripts of "1" by subscripts of "0". The covariance between student-level potential pretest and posttest scores within units is denoted by τ₀₁.²

Under this causal inference model, the difference between the two potential posttest scores, (Z_1Ti - Z_1Ci), is the unit-level treatment effect for unit i, and the average treatment effect parameter (ATE) = E(Z_1Ti - Z_1Ci) = μ_1T - μ_1C. The unit-level treatment effects, and hence, the ATE parameter, cannot be calculated directly because for each unit and student, the potential outcome is observed in either the treatment or control condition, but not in both. Formally, if T_i is a treatment status indicator variable that equals 1 for treatments and 0 for controls, then the observed posttest score for a unit, z_1i, can be expressed as follows:

Similarly, the observed posttest score for a student y_1ij is:

The simple equations in (4) and (5) form the basis for the causal inference theory presented below.

Importantly, (6) can also be derived using the following two-level HLM model (Bryk and Raudenbush 1992):

where Level 1 corresponds to students and Level 2 to units. Inserting the Level 2 equation into the Level 1 equation yields (6). Thus, the HLM approach is consistent with the causal inference theory presented above.³

A similar approach can be used to develop a regression model for the observed pretest scores:

where β₀= μ_0C, β₁= μ_0T - μ_0C, and μ_0i and e_0ij are between- and within-unit error terms, respectively, with the following properties: E(u_0i) = E(e_0ij) = 0; E(T_iu_0i) = E(T_ie_0ij) = 0; Var(u_0i) = σ₀²; Var(e_0ij) = τ₀²; Cov(u_0i, e_0ij) = 0; Cov(u_0i, u_1i) = σ₀₁; and Cov(e_0ij, e_1ij) = τ₀₁.

If the pretests are "true" baselines, β₁ will equal zero because of random assignment. Stated differently, with true baselines, Z_0Ti = Z_0Ci and Y_0Tij = Y_0Cij. With late baselines, the size and sign of β₁ will depend on the growth trajectory of intervention effects, the overall timing of baseline testing, and differences in testing-date distributions across the treatment and control groups. For example, β₁ will tend to be positive if the intervention has early beneficial intervention effects or if pretest testing dates are, on average, later for treatments than for controls.

Top