Co-Principal Investigator: Bell, Stephen

Randomized Controlled Trials (RCTs) are widely considered to provide the most convincing evidence on the impacts of education interventions. It can, however, be very difficult to conduct RCTs that have high external validity. Whatever the site selection mechanism — even if it is truly random — a nontrivial share of sites will decline to participate, and the remaining sites will typically be meaningfully less representative of the target population. This project seeks to improve the field’s capacity to conduct evaluations that produce findings which are externally valid or generalizable to broader policy settings by examining and developing methods to improve the external validity of randomized trials of education interventions when sites are not required to participate.

Using simulations, the Common Core of Data (CCD), and data from the Head Start Impact Study (HSIS), the research will test established methods for site selection: stratified random site selection and methods for systematically selecting sites to match the target population. The researchers will also develop and test methods for selecting replacement sites modelled after methods used in survey research. The research team will simulate simple purposive selection rules based on factors that researchers often consider when selecting sites. The team will then estimate external validity bias by comparing the impacts from the sites simulated to participate in the study to the impacts from all sites in the population. This will provide data with which to conduct a formal test for the presence of external validity bias and to estimate the mean squared error for each site selection method. Dissemination of results will include the usual journal manuscripts and conference presentations, but the research team also plans to meet with applied researchers who are preparing RCTs for IES and Education Innovation and Research (EIR) grants to discuss challenges and approaches to meeting those challenges for the sake of maintaining both internal and external validity. The grant team will then disseminate lessons learned from these interactions and recommendations for future evaluations.