Skip Navigation
Funding Opportunities | Search Funded Research Grants and Contracts

IES Grant

Title: Statistical Power When Adjusting for Multiple Hypothesis Tests: Methodology Expansions and Software Tools
Center: NCER Year: 2017
Principal Investigator: Porter, Kristin Awardee: MDRC
Program: Statistical and Research Methodology in Education      [Program Details]
Award Period: 3 years (7/1/17-06/30/20) Award Amount: $379,720
Type: Methodological Innovation Award Number: R305D170030

Multiplicity adjustments to p-values protect against spurious statistically significant findings when there are multiple statistical tests, but an important consequence of these adjustments is a change in statistical power. A recent paper by this grant's Principal Investigator, which is the product of an IES Early Career grant, presents a methodology for estimating power when accounting for multiplicity adjustments with one of five statistical procedures commonly used in education research: the Bonferroni, Holm, Benjamini-Hochberg, and single-step and stepdown versions of the Westfall-Young procedures. That paper, however, addresses only multiplicity from estimating effects on multiple outcomes using a simple design and analysis plan. The purpose of this grant is to extend that work to other designs, modeling assumptions, and analyses.

After mathematically extending the previous work to a broader range of designs, assumptions, and analyses, the research team will conduct Monte Carlo simulations to gauge the functioning and quality of the new equations. Following any necessary changes to and retesting of the equations, the research team will develop software for conducting the multiplicity adjustments. The team will also test the usability of the software and make changes as needed to ensure that it is user-friendly.

To disseminate the products from this research, the team will:

  • publish user-friendly, open-source software for applied researchers;
  • develop and make publicly accessible an interactive web application that allows users to create plots of power, minimum detectable effect sizes, or sample size requirements for studies with multiplicity;
  • create a webinar which will be available on MDRC's website; to develop a user's guide for applied researchers;
  • give conference presentations; and
  • publish manuscripts in peer-reviewed journals.

Related IES Project: Estimating Statistical Power in Impact Evaluations when Making Adjustments for Multiple Hypotheses (R305D140024)