An Empirical Analysis of Two Methodological Issues for Education Research Caused by Variation in Program Impacts

The purpose of the project is to investigate two issues pertinent to the variation in treatment effects across sites in a multisite trial. The first issue is the impact of cross-site variation on a study's statistical power to detect the mean treatment effect across all sites. While it is mathematically established how to account for such variation, there is little information available that helps researchers gauge the potential amount of the variation and thus the extent to which researchers need to compensate for it via other aspects of a design. The project team will use data from ten multisite randomized control trials of education interventions to estimate the cross-site variability in treatment effects. The findings from these data will provide a starting point for researchers who need to account for cross-site variability.

The second issue that researchers will address is individual-level variation in program effects. Regression discontinuity designs have strong internal validity at the cutpoint. Rather than assume that there is no validity at any other point, the research team will work to develop a way to gauge the potential validity at other points. Such generalization, however, is complicated to the extent that there is individual variation in treatment effects, so any new approach will need to take into account the individual-level variation.