Project Activities
Key outcomes
People and institutions involved
IES program contact(s)
Products and publications
ERIC Citations: Find available citations in ERIC for this award here or here.
Publicly Available Data:
- The project team developed an R package, 'generalizeR', which is available at Northwestern and will be submitted to CRAN.
Additional online resources and information:
- The project team developed a user-friendly webtool for planning randomized trials in education that integrates planning for generalizability and planning for statistical power to support the design of RCTs with both high internal and external validity. This free webtool is available at The Generalizer (www.thegeneralizer.org).
- Papers and resources related to the project can be found on the STEPP Center website at https://steppcenter.northwestern.edu/.
Select Publications:
Dong, N., Kelcey, B., & Spybrook, J. (2018). Power analyses for moderator effects in three-level cluster randomized trials. The Journal of Experimental Education, 86(3), 489-514.
Spybrook, J., Zhang, Q., Kelcey, B., & Dong, N. (2020). Learning from cluster randomized trials in education: An assessment of the capacity of studies to determine what works, for whom, and under what conditions. Educational Evaluation and Policy Analysis, 42(3), 354-374.
Tipton, E. (2021). Beyond generalization of the ATE: Designing randomized trials to understand treatment effect heterogeneity. Journal of the Royal Statistical Society Series A, 184(2), 504-521.
Tipton, E., Spybrook, J., Fitzgerald, K. G., Wang, Q., & Davidson, C. (2021). Toward a system of evidence for all: Current practices and future opportunities in 37 randomized trials. Educational Researcher, 50(3), 145-156.
Tipton, E. (2022). Sample selection in randomized trials with multiple target populations. American Journal of Evaluation, 43(1), 70-89.
Zhang, Q., Spybrook, J., & Unlu, F. (2020). Examining design and statistical power for planning cluster randomized trials aimed at improving student science achievement and science teacher outcomes. AERA Open, 6(3).
Additional project information
Previous award details:
Supplemental information
Co-Principal Investigator: Spybrook, Jessaca
- The project team developed new methods for designing studies including methods for planning for adequate power for tests of hypotheses regarding moderators, and methods for planning studies with multiple target estimands, such as multiple populations or moderator effects (Dong, Kelcey & Spybrook, 2018; Zhang, Spybrook, & Unlu, 2020; Tipton, 2022).
- The study team found the following results from the information collected on common practices regarding recruitment, research designs, and moderator analyses in efficacy studies in education from a sample of projects funded by IES between 2010 to 2015 (Spybrook, et. al. 2020; Titpon, et. al. 2021):
- Studies did not clearly report the population they were trying to represent or for whom their results would generalize.
- Most study samples did not represent populations of schools other than those in the district they fell within.
- Schools from large school districts were overrepresented relative to the population of public schools nationwide.
- The samples included were more homogeneous than the population.
- The research designs had adequate power for tests of individual level moderators but not school level moderators.
Questions about this project?
To answer additional questions about this project or provide feedback, please contact the program officer.
