Skip Navigation
Funding Opportunities | Search Funded Research Grants and Contracts

IES Grant

Title: Meta-analysis of Incorrectly Analyzed Studies
Center: NCER Year: 2016
Principal Investigator: VanHoudnos, Nathan Awardee: Carnegie Mellon University
Program: Statistical and Research Methodology in Education–Early Career      [Program Details]
Award Period: 2 years (9/2/2016- 9/30/2018) Award Amount: $199,918
Type: Methodological Innovation Award Number: R305D160016

Previous Grant Number: National Opinion Research Center (NORC)
Previous University Affiliation: R305D160009

The purpose of this project is to develop a correction strategy for significance tests of incorrectly analyzed studies (i.e., single-level regression was conducted instead of multilevel modeling). The correction strategy currently used by the What Works Clearinghouse (WWC) frequently leads to very low statistical power, leading to the potentially inaccurate conclusion that from a statistical significance standpoint, the tested interventions are ineffective. The researcher proposes to develop robust methods to quantify the uncertainty in the effect size that would have been estimated by the original authors if they had analyzed the study correctly. These new methods could then be employed by the WWC to give a more accurate assessment of the impact of interventions for which reported results were of the wrong analysis.

The researcher will first develop the theory for the new correction strategy. He will then create user-friendly code in R and in Stata for calculating the estimates and an Excel spreadsheet for conducting a meta-analysis with the estimates. He will demonstrate the new strategy and the corresponding software by conducting a meta-analysis of publicly available datasets. By the end of the project, he expects to have released a user-friendly version of the software and to be working on disseminating the results of the research at conferences and in peer-reviewed journals. The researcher also anticipates re-analyzing studies used in WWC Intervention Reports and preparing a paper illustrating discrepancies between the original conclusions and the conclusions resulting from the new approach.