|Title:||Statistical Power When Adjusting for Multiple Hypothesis Tests: Methodology Expansions and Software Tools|
|Principal Investigator:||Porter, Kristin||Awardee:||MDRC|
|Program:||Statistical and Research Methodology in Education [Program Details]|
|Award Period:||3 years (7/1/17-06/30/20)||Award Amount:||$379,720|
|Type:||Methodological Innovation||Award Number:||R305D170030|
Multiplicity adjustments to p-values protect against spurious statistically significant findings when there are multiple statistical tests, but an important consequence of these adjustments is a change in statistical power. A recent paper by this grant's Principal Investigator, which is the product of an IES Early Career grant, presents a methodology for estimating power when accounting for multiplicity adjustments with one of five statistical procedures commonly used in education research: the Bonferroni, Holm, Benjamini-Hochberg, and single-step and stepdown versions of the Westfall-Young procedures. That paper, however, addresses only multiplicity from estimating effects on multiple outcomes using a simple design and analysis plan. The purpose of this grant is to extend that work to other designs, modeling assumptions, and analyses.
After mathematically extending the previous work to a broader range of designs, assumptions, and analyses, the research team will conduct Monte Carlo simulations to gauge the functioning and quality of the new equations. Following any necessary changes to and retesting of the equations, the research team will develop software for conducting the multiplicity adjustments. The team will also test the usability of the software and make changes as needed to ensure that it is user-friendly.
To disseminate the products from this research, the team will:
Related IES Project: Estimating Statistical Power in Impact Evaluations when Making Adjustments for Multiple Hypotheses (R305D140024)