Purpose: This project extended and validated methods estimating power for different definitions of power and for several multiplicity adjustments commonly used in confirmatory analyses so that researchers who are planning randomized controlled trials (RCTs) with any number of common designs and who will adjust for multiple tests across multiple outcomes will be able to estimate power, minimum detectable effect sizes (MDESs), and sample sizes.

Project Activities: After mathematically extending the previous work to a broader range of designs, assumptions, and analyses, the research team conducted Monte Carlo simulations to gauge the functioning and quality of the new equations. The research team developed software for conducting the multiplicity adjustments. The team also tested the usability of the software and make changes as needed to ensure that it is user-friendly.

Key Outcomes: The key outcomes are reported in Hunter, K., Miratrix, L. and Porter, K. 2024 and summarized below.

• The team mathematically developed and validated methods (using Monte Carlo simulations) for estimating sample size, minimum detectable effect size (MDES), and statistical power, for
• multiple definitions of statistical power
• when applying any of five common multiplicity adjustments: Bonferroni, Holm, single-step and step-down versions of Westfall-Young, and Benjamini-Hochberg
• for multi-level research designs with one, two or three levels
• for multiple modeling choices - the model choices incorporate decisions about intercepts and treatment impacts:
• whether level two and level three intercepts are: fixed (a separate intercept for each unit) or random (a separate intercept for each unit as above, but model the collection of intercepts as Normally distributed, allowing for partial pooling)
• whether level two and level three treatment effects are: constant (model all units within a group as having the same single average impact), fixed (allow each block or cluster within a level to have its own individual estimated impact) or random (allow variation as with fixed, but model the collection of treatment impacts as Normally distributed around a grand mean impact).
• They applied these methods to produce empirical estimates of power, MDES and sample sizes through a worked example in a manuscript and a R package vignette based on the general design of MDRC's Diplomas Now evaluation, which contains three levels (students within schools within districts) with random assignment at level two (schools) (https://github.com/MDRCNY/pump_validate). These empirical estimates were used to make recommendations for applied researchers for both the design phase of their studies as well as when interpreting final results.
• They published open-source software, in the form of an R package with a set of functions for implementing the power estimation methods developed (https://cran.r-project.org/web/packages/PUMP/index.html).
• They also developed an interactive web application as an alternative to installing an R package (https://mdrc.shinyapps.io/pump/).

Related IES Project: Estimating Statistical Power in Impact Evaluations when Making Adjustments for Multiple Hypotheses( R305D140024)

Products and Publications

ERIC Citations: Find available citations in ERIC for this award here.

Select Publications:

Hunter, K., Miratrix, L. and Porter, K. (2024). PUMP: Estimating Power, Minimum Detectable Effect Size, and Sample Size When Adjusting for Multiple Outcomes in Multi-Level Experiments. Journal of Statistical Software, 108(6), doi: 10.18637/jss.v108.i06

Porter, K., Htet, Z., Hunter, K., Miratrix, L. (June 2022). The Power Under Multiplicity Project. Reflections on Methodology (Blog).

Software:

• R package: https://cran.r-project.org/web/packages/PUMP/index.html
• Package on Github: https://github.com/MDRCNY/PUMP
• Validation code and results on Github: https://github.com/MDRCNY/pump_validate

Applications:

• The PUMP app (web application): https://mdrc.shinyapps.io/pump/ along with a video illustrating how to use the app.