1. Introduction

Randomized control trials (RCTs) in the education field typically examine the intention-to-treat (ITT) parameter, which is estimated by comparing the mean outcomes of treatment group members (who are offered intervention services) to those of control group members (who are not). RCTs also sometimes examine two policy-relevant variants of the ITT parameter. The first variant is the complier average causal effect (CACE) parameter, defined as the average impact of intervention services on those who comply with their treatment assignments (Bloom 1984; Angrist et al. 1996). Estimators for this parameter are obtained by adjusting the ITT impact estimators for those in the treatment group who do not receive intervention services and for crossovers in the control group who erroneously receive intervention services. Second, it is becoming increasingly popular to standardize ITT and CACE impact estimates into effect size (standard deviation) units. This metric is useful for comparing findings across outcomes that are measured on different scales, for interpreting impacts that are difficult to understand in nominal units, and for comparing study findings to those from previous evaluations (Cohen 1988; Lipsey and Wilson 1995; Hedges 1981 and 2007).

This report addresses two main issues. First, it systematically examines the identification of the CACE parameter under clustered RCT designs that are typically used in the education field, where units (such as schools or classrooms) rather than students are randomly assigned to a treatment or control condition. Using a causal inference and instrumental variables (IV) framework, we extend the identification conditions in Angrist et al. (1996) to two-level clustered designs, where treatment compliance decisions can be made by both school staff and students. Our emphasis differs from Jo et al. (2008) who focus on parametric and path modeling of treatment noncompliance under clustered designs using multilevel mixture models and maximum likelihood methods.

The second purpose of the report is to theoretically and empirically examine variance estimation under clustered designs for two types of IV estimators: (1) CACE estimators in nominal units, and (2) ITT and CACE estimators in effect size units—hereafter referred to as standardized estimators. These estimators are ratio estimators, whose variances must account for estimation errors in their numerators and denominators. In practice, however, analysts often ignore the estimation error in the denominator terms, which are assumed to be known. Thus, in study reports, the same t-statistics and p-values are sometimes reported for all estimators.

A potential problem with this approach, however, is that it could lead to significance findings that are biased if the variance correction terms for the denominators matter. Accordingly, we present simple asymptotic variance estimation formulas for commonly-used ratio estimators by combining variance results in Hedges (2007) for standardized ITT estimators with those in Little et al. (2006) and Heckman et al. (1994) for CACE estimators. We then use data from ten large-scale RCTs to compare significance findings using the correct variance formulas with those that are typically used in practice, an empirical issue that has not been systematically addressed in the literature. The empirical results can be used to help guide future decisions as to whether the correct, but more complex variance formulas are warranted for RCTs in the education area to obtain rigorous significance findings for the full range of impact estimators.

The remainder of this report is in six chapters. Chapter 2 discusses the causal inference framework underlying the ITT estimator for two-level clustered designs, which forms the foundation for the CACE analysis. Chapter 3 discusses impact and variance estimation of the ITT parameter, and Chapter 4 discusses identification and estimation of the CACE parameter. Chapter 5 discusses estimation of the impact parameters in effect size units. Chapter 6 discusses empirical findings, and the final chapter presents a summary and conclusions.

Top