NCEE Technical Methods Report: Do Typical RCTs of Education Interventions Have Sufficient Statistical Power for Linking Impacts on Teacher Practice and Student Achievement Outcomes (October 13, 2009)
Reports in this series are designed for use by researchers, methodologists, and evaluation specialists to provide guidance in resolving or advancing challenges to evaluation methods. For RCTs of education interventions, it is often of interest to estimate associations between student and mediating teacher practice outcomes, to examine the extent to which the study’s conceptual model is supported by the data, and to identify specific mediators that are most associated with student learning.
What to Do When Data Are Missing in Group Randomized Controlled Trials (October 13, 2009)
This NCEE Technical Methods report examines how to address the problem of missing data in the analysis of data in Randomized Controlled Trials (RCTs) of educational interventions, with a particular focus on the common educational situation in which groups of students such as entire classrooms or schools are randomized. Missing outcome data are a problem for two reasons: (1) the loss of sample members can reduce the power to detect statistically significant differences, and (2) the introduction of non-random differences between the treatment and control groups can lead to bias in the estimate of the intervention's effect. The report reviews a selection of methods available for addressing missing data, and then examines their relative performance using extensive simulations that varied a typical educational RCT on three dimensions: (1) the amount of missing data; (2) the level at which data are missing—at the level of whole schools (the assumed unit of randomization) or for students within schools; and, (3) the underlying missing data mechanism. The performance of the different methods is assessed in terms of bias in both the estimated impact and the associated standard error. Reports in this series are designed for use by researchers, methodologists, and evaluation specialists to provide guidance in resolving or advancing challenges to evaluation methods.
