Chapter 4: The CACE Parameter

The ITT estimator provides information on treatment effects for those in the study population who were offered intervention services. The treatment group sample used to estimate this parameter, however, might include not only students who received services but also those who did not. Similarly, the control group sample may include crossovers who received embargoed intervention services for advertent or inadvertent reasons. In these cases, the ITT estimates may understate intervention effects for those who were eligible for and actually received services (assuming that the intervention improves outcomes). Thus, it is often of policy interest to estimate the CACE parameter that pertains to those who complied with their treatment assignments.

It is important to recognize that if treatment group noncompliers existed in the evaluation sites, they are likely to exist if the intervention were implemented more broadly. Thus, the ITT parameter pertains to real-world treatment effects. The CACE parameter, however, is important for understanding the “pure” effects of the intervention for those who received meaningful intervention services, especially for efficacy studies that aim to assess whether the studied intervention can work. Decision makers may also be interested in the CACE parameter if they believe that intervention implementation could be improved in their sites. Furthermore, the CACE parameter can be critical for drawing policy lessons from ITT effects; for instance, the CACE parameter can distinguish whether a small ITT effect is due to low rates of compliance or due to small treatment effects among compliers, with each scenario implying different strategies for improving intervention effects.

Sources of Noncompliance

Under clustered RCT designs in the education area, the extent to which students receive intervention services could depend on compliance decisions made by both school staff (such as superintendents, principals, and teachers) and students. The interplay between these sources will depend on the particular intervention and study design (Jo et al. 2008). Furthermore, the extent of compliance will depend on the approach for defining service dosage, which is a topic that is beyond the scope of this report. For context, however, in what follows, we briefly discuss general sources of noncompliance at the school and student levels.

Noncompliance by School Staff
School staff in treatment units may not offer intervention services for several reasons. First, school principals or district superintendents may change their minds about implementing the intervention, due to changes in school priorities or for other reasons. Second, even if schools agree to participate, some teachers may not, perhaps due to initial problems implementing the intervention or because they prefer their status quo teaching methods or curricula. In addition, noncompliance could occur if school personnel are not adequately trained in intervention procedures. Similarly, crossovers could occur if staff in control schools decide to offer the intervention (or a very similar one), perhaps because of a strong belief that the intervention is effective (from discussions with evaluators) and a strong desire to implement it immediately rather than after the embargo period.

Noncompliance by Students
Students may also play a role in noncompliance for several reasons. First, a student may not receive meaningful intervention services due to a lack of school attendance. This could occur, for example, if the student is suspended, is chronically absent, or, if relevant, decides not to attend a voluntary program (for example, an after-school program).

Second, student mobility in and out of the study schools could lead to a low dosage of service receipt. In some designs, follow-up data are collected only for students who are present in the study schools at baseline (to ensure that the treatment and control group student samples will have similar baseline characteristics). In these designs, noncompliers may include those who left the treatment schools soon after the start of the school year. A more common “placed-based” design, however, is when follow-up data are collected for all students in the target grades who are in the study schools at data collection, including those who entered the schools after baseline. In these designs, noncompliers could include students who entered the study schools soon before follow-up data collection.¹ Under either design, crossovers could occur due to student mobility if control students in the follow-up sample transfer to treatment schools or classrooms.

Identification of the CACE Parameter

This section discusses the identification of the CACE parameter under two scenarios. First, to fix ideas, we assume that compliance is determined solely by school staff, and that all students who are offered services receive them. Second, we consider the more general case where compliance is determined by both schools and students, in which case some students may not receive services even if their schools offer them. For both scenarios, treatment status (T_i) is determined at random assignment and is fixed thereafter; T_i values are not affected by compliance decisions. We assume also that if the RCT uses a “placed-based” design as discussed above, there are no treatment effects on student mobility. Finally, because the literature has conceptualized compliance decisions as dichotomous (Angrist et al. 1996), we model the offer and receipt of services as binary decisions.

In what follows, we introduce some new notation. Let R_i =R_i(T_i) denote an indicator variable that equals 1 if unit i would offer intervention services if assigned to a given treatment condition (T_i =0 or T_i =1), and let W_i(T_i, R_i) denote the unit's potential outcome for a given value of (T_i, R_i); there are four such potential outcomes. Similarly, let D_ij =D_ij (T_i, R_i) denote an indicator variable that equals 1 if the student receives intervention services from any study school, given one of the four possible combinations of (T_i, R_i). Finally, let Y_ij (T_i, R_i, D_ij) denote the student's potential outcome, given one of the possible combinations of (T_i, R_i, D_ij).

The CACE Parameter When Compliance Decisions Are Made by Units Only
To identify the CACE parameter when treatment compliance decisions are made by units only, we classify units into four mutually exclusive compliance categories: compliers, never-takers, always-takers, and defiers (Angrist et al. 1996). Compliers (CL) are those who would offer intervention services only if they were assigned to the treatment group [R_i(1)=1 and R_i(0)=0]. Never-takers (N) are those who would never offer treatment services [R_i(1)=0 and R_i(0)=0], and always-takers (A) are those would always offer treatment services [R_i(1)=1 and R_i(0)=1]. Finally, defiers (D) are those who would offer treatment services only in the control condition [R_i(1)=0 and R_i(0)=1]. Outcome data are assumed to be available for all sample members. Note that this scenario applies also to nonclustered designs where units are students.

The ITT parameter for the pooled sample α_ITT can be expressed as a weighted average of the ITT parameters for each of the four unobserved compliance groups:

where p_g is the fraction of the study population in compliance group g(Σ p_g =1), and α_{ITT_g} is the associated ITT impact parameter (as defined earlier).

Following Angrist et al. (1996), the α_{ITT_CL} parameter in (15) can then be identified under three key assumptions (U1-U3):

U1. The Unit-Level Stable Unit Treatment Value Assumption (SUTVA): Unit-level potential compliance decisions [R_i(T_i)] and outcomes [W_i(T_i, R_i)] are unrelated to the treatment status of other units. This allows us to express R_i(T_i) and W_i(T_i, R_i) in terms of T_i rather than the vector of treatment statuses of all units. This condition is likely to hold in clustered education RCTs where random assignment is conducted at the school level (the most common design), unless there is substantial interaction between students and staff across study schools.

U2. Unit-Level Monotonicity: R_i(1) ≥R_i(0) . This means that units are at least as likely to offer intervention services in the treatment than control condition, and implies that there are no defiers (that is, p_D =0). Under this assumption, p_CL =P(R_i(1) =1) -P(R_i(0) =1) which is the difference between service offer rates in the treatment and control conditions.

U3. The Unit-Level Exclusion Restriction: W_i(1,r) =W_i(0,r) for r = 0,1. This means that the outcome for a unit that offers services would be the same in the treatment or control condition, and similarly for a unit that does not offer services. Stated differently, this restriction implies that any effect of T_i on outcomes must occur only through an effect of T_i on service offer rates. This restriction implies that impacts on always-takers and never-takers are zero, that is, α_{ITT_N} =α_{ITT_A} =0.

Under these assumptions, the final three terms on the right-hand-side of (15) cancel. Thus, the following CACE impact parameter can be identified:

This parameter represents the average causal effect of the treatment for compliers.

Importantly, follow-up data on all sample members are required to estimate the CACE parameter even though this parameter pertains to the complier subgroup only. Thus, noncompliance is different than data nonresponse.

The CACE Parameter When Compliance Decisions Are Made by Units and Students
In this section, we generalize the CACE parameter from above to the case where compliance decisions are made by both school staff and students. For this analysis, we require assumptions on both students and schools to identify the CACE parameter.

Table 4.1 displays and labels the 16 possible student-level complier groups that depend on treatment status (T_i), whether the school offers services (R_i), and whether the student receives services (D_ij). In this scenario, there are four groups each of compliers, never-takers, defiers, and always-takers. For example, Never-Taker Group 2 includes students who would never receive services even though their schools would always offer them. Note that students with R_i =0 and D_ij =1 are assumed to receive services from a different study school than their baseline school. The frequency of each of the 16 combinations will depend on the particular application, and some may be rare. However, all combinations are included for completeness.

To derive the CACE parameter under this scenario, we define α_ITTⁱ =E(Y_Tij -Y_cij | W_Ti, W_Ci, p_1i, ...,p_16i) as the within-unit ITT for the student population in unit i, where p_gi is the fraction of students in compliance group Note that E(α_ITTⁱ) =α_ITT where the expectation is taken over the joint unit-level potential outcome and compliance distributions. Note next that α_ITTⁱ can be expressed as a weighted average of the within-unit ITT parameters for each of the 16 student-level compliance groups shown in Table 4.1:

where α_{ITT_g}ⁱ is the impact parameter for compliance group g.

Using (17) and Table 4.1, the CACE parameter for Complier Group 1 can be identified under the following assumptions (that are analogs to the unit-level assumptions from above):

S1. SUTVA: Potential student-level service receipt decisions [D_ji(T_i, R_i)] and outcomes [Y_ij(D_ji, T_i, R_i)] are unrelated to the treatment status of other students and schools. In addition, we impose the unit-level SUTVA condition U3 from above that R_i(T_i) is unrelated to the treatment status of other units.

S2. Monotonicity on Compliance: R_i(1) ≥R_i(0) or D_ij(1, R_i(1)) ≥D_ij(0, R_i(0)) . This assumption will be satisfied if a unit is at least as likely to offer services in the treatment than control condition, or if students in that unit are at least as likely to receive services in the treatment than control condition. Using Table 4.1, this condition implies that p₁₆ =0.

S3. Student-Level Monotonicity on the Take-Up of Services: D_ij(s, 1) ≥D_ij(t, 0) for s,t∈{0,1}. This assumption means that students are at least as likely to take up services if they are offered them than if they are not, which implies that p₆ =p₁₁ =0.

S4. The Student-Level Exclusion Restriction on Compliance: D_ij(1,r) =D_ij(0,r) for r =0,1. This means that for a given service offer status, the student's compliance decision would be the same in the treatment or control condition. Stated differently, this restriction implies that any effect of T_i on student compliance decisions must be a result of a treatment effect on service offer rates. Using Table 4.1, this restriction implies that p₃ =p₈ =p₉ =p₁₄ =0.

S5. The Student-Level Exclusion Restriction on Outcomes: Y_ij (1,R_i(1), d) =Y_ij (0,R_i(0), d) for d =0,1. This means that student outcomes are determined solely by whether or not the student receive services, and it does not matter where these services are received (or not received) or how many other students are receiving them. This restriction implies zero impacts for Groups 2, 4, 5, 7, 10, 12, 13, and 15.

These assumptions imply that the only term on the right-hand-side of (17) that does not cancel is the first term that pertains to Complier Group 1 (see Table 4.1). Thus, after taking expectations in (17), the following CACE parameter can be identified:

This CACE parameter is a weighted average of within-unit impacts for Complier Group 1, with weights p_1i (and reduces to E(α_{ITT_1}ⁱ if α_{ITT_1}ⁱ and p_1i are independent). Denoting p_g =E(p_gi), assumptions S2 to S4 imply that p₁ =(p_T -p_C) where P_T =p₁ +p₂ p₄ p₁₀ p₁₂ and p_C =p₂ +p₄ +p₁₀ +p₁₂ are the fractions of students receiving services in the treatment and control conditions, respectively. Thus, the only difference between α_CACE0 in (16) and α_CACE in (18) is that p_CL refers to service offer rates for units whereas p₁ refers to service receipt rates for students. Clearly, (18) is more general and reduces to The CACE Parameter 15 (16) if compliance decisions are made by school staff only. Thus, in what follows, we focus on estimation issues for the α_CACE parameter.₂

Impact and Variance Estimation of the CACE Parameter

In this section, we discuss estimation of the CACE parameter. We use an IV approach, because simple closed-form variance formulas exist, the variance correction terms can easily be understood because they enter the formulas linearly, and the formulas can be readily generalized to the standardized CACE parameter. An alternative approach, without these properties, is to use (more efficient) maximum likelihood estimation methods and the EM algorithm (Jo et al. 2008).

Impact Estimation
A consistent estimator for α_CACE in (18) can be obtained by dividing consistent estimators for α_ITT and p₁:

Estimators for p₁ =p_T -p_C can be obtained by noting that this parameter represents an impact on the rate of service receipt. Thus, estimation methods similar to those discussed above for α_ITT can be used to estimate p₁. For example, analogous to (5), the simple differences-in-means estimator is where d_ij is an observed service receipt status indicator variable that equals 1 if student i in school j received intervention services, and zero otherwise.

Variance Estimation
The CACE estimator in (19) is a ratio estimator (Little et al. 2008 and Heckman et al. 1994). Both the numerator and denominator are measured with error, and thus, both sources of error should be taken into account in the variance calculations. A variance estimator for α̂_CACE can be obtained using an asymptotic Taylor series expansion of α̂_CACE around the true value α_CACE:

Taking squared expectations on both sides of (20) and inserting estimators for unknown parameters yields the following variance estimator for α̂_CACE:

The first term in (21) is the variance of the CACE estimator assuming that estimated service receipt rates are measured without error. The second and third terms are therefore correction terms. The second term accounts for the estimation error in p̂₁, and the third term accounts for the covariance between α̂_ITT and p̂₁.³ Importantly, these correction terms depend on the size of α_ITT and thus, become more important with larger impacts. Finally, because α̂_ITT and p̂₁ are asymptotically normal, the delta method (Greene 2000, p.118) implies that α̂_CACE is also asymptotically normal.

An asymptotic variance estimator for p̂₁ that adjusts for clustering can be obtained from (10) and (11) using simple differences-in-means methods or from (13) and (14) using linear probability models, where y_ij is replaced by d_ij. For our empirical work, we used a slightly different variance estimator that allows for different processes underlying service receipt decisions for treatments and controls:

where , and k is the number of unit-level covariates (including the intercept) that are included in the model. Similarly, an unbiased estimator for 1 AsyCov(α̂_ITT, p̂₁) is as follows:

where .

Finally, the CACE impact and variance estimators discussed above are IV estimators (Angrist et al. 1996). To see this, consider the following variant of the model in (3):

where u_i^IV and e_ij^IV are random error terms. If T_i is used as an instrument for d_ij in (24), then the estimated IV regression coefficient α̂_CACE is the simple differences-in-means CACE estimator in (19) with the variance estimator in (21). Treatment status T_i is likely to be a "strong" instrument if service receipt rates differ markedly for treatment and control students (see Murray 2006 and Stock et al. 2002 for a discussion of weak and strong instruments).⁴

Top