Previous Grant Number: National Opinion Research Center (NORC)
Previous University Affiliation: R305D160009

Purpose: The purpose of this project was to develop a correction strategy for significance tests of incorrectly analyzed studies (i.e., single-level regression was conducted instead of multilevel modeling). The correction strategy currently used by the What Works Clearinghouse (WWC) frequently leads to very low statistical power, leading to the potentially inaccurate conclusion that from a statistical significance standpoint, the tested interventions are ineffective. The researcher developed robust methods to quantify the uncertainty in the effect size that would have been estimated by the original authors if they had analyzed the study correctly. These new methods could then be employed by the WWC to give a more accurate assessment of the impact of interventions for which reported results were of the wrong analysis.

Project Activities: The researcher first developed the theory for the new correction strategy. He will then create user-friendly code in R and in Stata for calculating the estimates and an Excel spreadsheet for conducting a meta-analysis with the estimates. He demonstrated the new strategy and the corresponding software by conducting a meta-analysis of publicly available datasets. By the end of the project, he released a user-friendly version of the software and was disseminating the results of the research at conferences and in peer-reviewed journals. The researcher also anticipated re-analyzing studies used in WWC Intervention Reports and preparing a paper illustrating discrepancies between the original conclusions and the conclusions resulting from the new approach.