Chapter 7: Summary and Conclusions

This report has examined the identification and estimation of the CACE parameter for two-level clustered RCTs that are commonly used in education research, where groups (such as schools or classrooms) rather than students are the unit of random assignment. We generalized the causal inference and IV framework developed by Angrist et al. (1996) to develop conditions for identifying the CACE parameter under clustered designs where multi-level treatment compliance decisions can be made by both school staff and students.

This report also provides simple asymptotic variance estimation formulas for CACE impact estimators measured in both nominal and standard deviation units. Because these IV impact estimators are ratio estimators, the variance formulas account for both the estimation error in the numerators (which pertain to the nominal ITT impact estimates) and the denominators (which pertain to the estimated service receipt rates and the estimated standard deviations of the outcomes).

Researchers sometimes assume that the denominator terms in these ratio estimators are known, and thus, present the same p-values from significance tests for all ITT and CACE impact estimates. This approach, however, could yield incorrect significance findings if the variance components due to the denominator terms matter. Accordingly, we used data from 10 large-scale RCTs in education and other social policy areas to compare significance findings for the considered impact estimates using uncorrected and corrected variance estimators.

Our key empirical finding is that the variance correction terms have very lITTle effect on the standard errors of the standardized ITT and CACE impact estimators. Across the examined outcomes, the correction terms typically raise the standard errors by less than 1 percent, and change p-values at the fourth or higher decimal place. Furthermore, simulations indicate that, on average, the impact estimates would need to be 0.7 to 0.8 standard deviations, representing effect sizes that are rarely found in practice, before the variance corrections would raise the standard errors by 5 percent. These results occur because, by far, the most important source of variance in the considered ratio estimators is the variance of the nominal ITT impact estimators.

Despite these results, we advocate, for rigor, that education researchers use the correct standard error formulas for standardized ITT and CACE impact estimates. The formulas laid out in this report are relatively straightforward to apply, and their use will protect against the risk of finding incorrect significance findings, even if this risk is likely to be low based on our empirical findings.

Top