Grant Closed

Estimating Statistical Power in Impact Evaluations when Making Adjustments for Multiple Hypotheses

NCER

Program: Statistical and Research Methodology in Education

Program topic(s): Early Career

Award amount: $199,845

Principal investigator: Kristin Porter

Awardee:

MDRC

Year: 2014

Award period: 1 year 6 months (09/01/2014 - 02/29/2016)

Project type:

Methodological Innovation

Award number: R305D140024

Purpose

Multiplicity adjustments to p-values protect against spurious statistically significant findings when there are multiple statistical tests (e.g., due to multiple outcomes, subgroups or time points). An important consequence of these adjustments is a change in statistical power. It is typically argued that multiplicity adjustments result in a loss of power, which can be substantial. Current practice for ensuring that impact evaluations in education have adequate statistical power does not take the use of multiplicity adjustments into account. This project investigated alternatives to current practice in research studies that adjust for multiplicity. This project also investigated alternatives to standard practice for how power is defined in studies that adjust for multiplicity. The education literature that discusses the power consequences of using multiple testing procedures focuses on the power of each individual test, but this is not always the best approach to conceptualizing statistical power.

Project Activities

Researchers conducted research via analytic extensions of existing procedures in other fields (i.e., medicine, genomics, and biostatistics) and simulation studies. Researchers developed a practice guide that provides applied researchers with step-by-step instructions for implementing the set of methods developed for calculating statistical power, as well as example computer code for obtaining the calculations. The project’s impacts on future research include the potential for more accurate estimates of power (or of minimum detectable effect size or of sample size for a given power requirement) and the potential for more appropriate estimates of power than those that are currently used.