Project Activities
After mathematically extending the previous work to a broader range of designs, assumptions, and analyses, the research team conducted Monte Carlo simulations to gauge the functioning and quality of the new equations. The research team developed software for conducting the multiplicity adjustments. The team also tested the usability of the software and make changes as needed to ensure that it is user-friendly.
Key outcomes
The key outcomes are reported in Hunter, K., Miratrix, L. and Porter, K. 2024 and summarized below.
People and institutions involved
IES program contact(s)
Products and publications
ERIC Citations: Find available citations in ERIC for this award here.
Select Publications:
Hunter, K., Miratrix, L. and Porter, K. (2024). PUMP: Estimating Power, Minimum Detectable Effect Size, and Sample Size When Adjusting for Multiple Outcomes in Multi-Level Experiments. Journal of Statistical Software, 108(6), doi: 10.18637/jss.v108.i06
Porter, K., Htet, Z., Hunter, K., Miratrix, L. (June 2022). The Power Under Multiplicity Project. Reflections on Methodology (Blog).
Software:
- R package: https://cran.r-project.org/web/packages/PUMP/index.html
- Package on Github: https://github.com/MDRCNY/PUMP
- Validation code and results on Github: https://github.com/MDRCNY/pump_validate
Applications:
- The PUMP app (web application): https://mdrc.shinyapps.io/pump/ along with a video illustrating how to use the app.
Related projects
Supplemental information
- The team mathematically developed and validated methods (using Monte Carlo simulations) for estimating sample size, minimum detectable effect size (MDES), and statistical power, for
- multiple definitions of statistical power
- when applying any of five common multiplicity adjustments: Bonferroni, Holm, single-step and step-down versions of Westfall-Young, and Benjamini-Hochberg
- for multi-level research designs with one, two or three levels
- for multiple modeling choices - the model choices incorporate decisions about intercepts and treatment impacts:
- whether level two and level three intercepts are: fixed (a separate intercept for each unit) or random (a separate intercept for each unit as above, but model the collection of intercepts as Normally distributed, allowing for partial pooling)
- whether level two and level three treatment effects are: constant (model all units within a group as having the same single average impact), fixed (allow each block or cluster within a level to have its own individual estimated impact) or random (allow variation as with fixed, but model the collection of treatment impacts as Normally distributed around a grand mean impact).
- They applied these methods to produce empirical estimates of power, MDES and sample sizes through a worked example in a manuscript and a R package vignette based on the general design of MDRC's Diplomas Now evaluation, which contains three levels (students within schools within districts) with random assignment at level two (schools) (https://github.com/MDRCNY/pump_validate). These empirical estimates were used to make recommendations for applied researchers for both the design phase of their studies as well as when interpreting final results.
- They published open-source software, in the form of an R package with a set of functions for implementing the power estimation methods developed (https://cran.r-project.org/web/packages/PUMP/index.html).
- They also developed an interactive web application as an alternative to installing an R package (https://mdrc.shinyapps.io/pump/).
Questions about this project?
To answer additional questions about this project or provide feedback, please contact the program officer.
