Chapter 5: The Standardized ITT and CACE Estimators

It is becoming increasingly popular in educational research to standardize estimated impacts into standard deviation units (Hedges 1981 and 2007). This approach can be used to facilitate the comparison of impact findings across outcomes that are measured on different scales. It has also been used extensively in metaanalyses to contrast and collate impact findings across a broad range of disciplines (Cohen 1988; Lipsey and Wilson 1993). The use of effect sizes is especially important for helping to understand impact findings on outcomes that are difficult to interpret when measured in nominal units (for example, impacts on behavioral scales or test scores). In addition, this approach is useful for creating composite measures across multiple outcomes, and for scaling an outcome that is measured differently across students (such as state achievement test scores from different states). Finally, it has become standard practice in education evaluations to conduct power analyses using primary outcomes that are measured in effect size units, to ensure adequate study sample sizes for detecting impacts that are meaningful and attainable based on findings from previous studies.

Impact Estimation for the Standardized ITT Estimator

The ITT parameter in effect size units, α_{ITT_E}, can be expressed as follows:

where σ_y is the standard deviation of the outcome across all treatment and control students.⁵ An unbiased standard deviation estimator for can be obtained as follows:

where S_B² and S_W² are defined as in (9) and (7) above. Thus, a consistent estimator for α_{ITT_E} is:

Variance Estimation for the Standardized ITT Estimator

The effect size estimator in (27) is a ratio estimator where both the numerator and denominator are measured with error. We discern, however, two competing views on whether it is necessary, when reporting impact results, to adjust the variance of this estimator for the estimation error in S_y. One view, that opposes variance corrections, is that standardized impact estimators are descriptive statistics for interpreting and benchmarking the impacts in nominal units. In this view, standardized outcomes are not measures per se, and thus, the nominal estimator is the relevant impact for assessing whether an education intervention had a significant impact on the outcome. The alternative view is that the standardized impact estimator is often the impact measure on which researchers and policymakers focus. Thus, standardized outcomes are effectively the outcome measures of interest, and standardized impacts should have proper standard errors attached to them.

Given these opposing views, we believe that it is appropriate that impact studies report correct standard errors for ITT impact estimates in both nominal and effect size units. Thus, in what follows, we discuss simple asymptotic variance formulas for the standardized estimators (see Hedges 2007 for similar results using finite populations and unequal cluster sizes).

A variance estimator for α̂_{ITT_E} can be obtained from the delta method using a Taylor series expansion of α̂_{ITT_E} around the true value α_{ITT_E}, which after inserting estimators for unknown parameters, yields the following expression:

where the asymptotic covariance term between α̂_ITT and S_y can be shown to be zero using results on the independence of linear functions and quadratic forms for normal distributions.

The first term in (28) is the variance expression for the effect size impact ignoring the estimation error in S_y (the usual approach found in the literature). The second term, therefore, is a correction term. This term increases as α̂_ITT increases, and is zero if and only if α̂_ITT =0.

Finally, (28) requires an estimator for AsyVar(S_y), which can be obtained as follows (see Appendix A for a proof):

This expression also applies to nonclustered designs where units are defined as students. In this case, S_W² = 0 and S_y² = S_B² so that (29) reduces to S_B² /[2(n -2)].

Impact and Variance Estimation for the Standardized CACE Estimator

Using results from above, a CACE estimator in effect size units can be expressed as follows:

where it is assumed that the standard deviation for compliers is the same as it is for the full sample. Using the delta method, a variance estimator for α̂_{CACE_E} is:

where we have ignored the covariance term between S_y and p̂₁. The estimator α̂_{CACE_E} is asymptotically normal because each estimator component is asymptotically normal.

Top