Purpose: Multiplicity adjustments to p-values protect against spurious statistically significant findings when there are multiple statistical tests (e.g., due to multiple outcomes, subgroups or time points). An important consequence of these adjustments is a change in statistical power. It is typically argued that multiplicity adjustments result in a loss of power, which can be substantial. Current practice for ensuring that impact evaluations in education have adequate statistical power does not take the use of multiplicity adjustments into account. This project investigated alternatives to current practice in research studies that adjust for multiplicity. This project also investigated alternatives to standard practice for how power is defined in studies that adjust for multiplicity. The education literature that discusses the power consequences of using multiple testing procedures focuses on the power of each individual test, but this is not always the best approach to conceptualizing statistical power.

Project Activities: Researchers conducted research via analytic extensions of existing procedures in other fields (i.e., medicine, genomics, and biostatistics) and simulation studies. Researchers developed a practice guide that provides applied researchers with step-by-step instructions for implementing the set of methods developed for calculating statistical power, as well as example computer code for obtaining the calculations. The project’s impacts on future research include the potential for more accurate estimates of power (or of minimum detectable effect size or of sample size for a given power requirement) and the potential for more appropriate estimates of power than those that are currently used.

Products and Publications

Journal article, monograph, or newsletter

Porter, K. E. (2018). Statistical Power in Evaluations That Investigate Effects on Multiple Outcomes: A Guide for Researchers. Journal of Research on Educational Effectiveness, 11(2), 267–295.