Project Activities
The project team carried out two sets of activities. First, the team compared the performance of different methods for selecting sites—and replacing sites that refuse to participate—in controlling external validity bias randomized trials of educational interventions. Second the team developed and disseminated computer code and other resources that other researchers can use to implement different sampling and analysis methods for improving the generalizability of randomized trials and other impact studies.
Structured Abstract
Statistical/Methodological Product: The research team provided results from testing the performance of different methods for selecting more representative samples of districts and schools for impact studies with a focus on (1) random site selection and (2) balanced site selection designed to minimize differences in characteristics between the sample and the population. They also developed and released software to support each type of site selection.
Development/Refinement Process: The study involved simulation research using data from the Common Core of Data and the Head Start Impact Study. From these data, repeated samples were selected using various site selection and replacement methods. The external validity bias of each site selection method was calculated by subtracting the population average treatment effect from the mean sample average treatment effect across the simulated samples.
Key outcomes
- The research team published details on the external validity bias that results from different approaches to selecting sites (e.g., districts and schools) and replacing sites that decline to participate in a peer reviewed article. Their study found evidence that both random and balanced site selection yields impact estimates with less external validity bias and lower mean squared error than purposive site selection, even when a large share of sites decline to participate (Litwok., Shivji, and Olsen 2023).
- Software (an R package and two SAS macros) to implement different approaches (random and balanced site selection) to selecting and replacing sites were developed and released (https://www.generalizability.org/).
People and institutions involved
IES program contact(s)
Project contributors
Products and publications
ERIC Citations: Find available citations in ERIC for this award here.
Additional Online Resources and Information:
- https://cran.r-project.org/web/packages/sitepickR/index.html - an R package for selecting districts and schools using the cube method
- https://github.com/select-sites/exval - a SAS macro for stratifying a population frame of schools using cluster analysis and a SAS macro for selecting stratified random and stratified balanced samples of districts and schools
Select Publications:
Litwok, D., Nichols, A., Shivji, A., & Olsen, R. B. (2023). Selecting districts and schools for impact studies in education: A simulation study of different strategies. Journal of Research on Educational Effectiveness, 16(3), 501-531.
Project website:
Related projects
Questions about this project?
To answer additional questions about this project or provide feedback, please contact the program officer.
