Statistical Methods for Using Rigorous Evaluation Results to Improve Local Education Policy Decisions

The purpose of this project is to investigate methods for testing the generalizability of the results of a multi-site study to sites that were not in the study's sample. Results of a well-designed and well-implemented cluster-randomized trial of a new curriculum, for example, may accurately reflect the average effect of the tested curriculum in the evaluated sites and even the average impact it would have in the population. Any individual school district, however, is likely to differ materially from the average evaluation site, possibly in ways that would affect the impact of the new curriculum. The average results of the evaluation might therefore be misleading for any given site.

Using four real data sets and using simulations, the research team will test multiple approaches to predicting site-specific impacts from full-study results. The approaches they will investigate are: using the overall average impact; using subgroup-specific impact estimates; response surface modeling; and two ways of reweighting the original sample (via propensity score reweighting and kernel weighting) to make it like the target site. After comparing the performance of these approaches, the team will begin to disseminate the results in seminars, conference presentations, and peer-reviewed journal manuscripts. Depending on the computational complexity of the favored approaches, the researchers may also create and disseminate software to run the analyses.