Skip to main content

Breadcrumb

Home arrow_forward_ios Information on IES-Funded Research arrow_forward_ios Meta-analysis of Incorrectly Analyz ...
Home arrow_forward_ios ... arrow_forward_ios Meta-analysis of Incorrectly Analyz ...
Information on IES-Funded Research
Grant Closed

Meta-analysis of Incorrectly Analyzed Studies

NCER
Program: Statistical and Research Methodology in Education
Program topic(s): Early Career
Award amount: $199,918
Principal investigator: Nathan VanHoudnos
Awardee:
Carnegie Mellon University
Year: 2016
Project type:
Methodological Innovation
Award number: R305D160016

Purpose

The purpose of this project was to develop a correction strategy for significance tests of incorrectly analyzed studies (i.e., single-level regression was conducted instead of multilevel modeling). The correction strategy currently used by the What Works Clearinghouse (WWC) frequently leads to very low statistical power, leading to the potentially inaccurate conclusion that from a statistical significance standpoint, the tested interventions are ineffective. The researcher developed robust methods to quantify the uncertainty in the effect size that would have been estimated by the original authors if they had analyzed the study correctly. These new methods could then be employed by the WWC to give a more accurate assessment of the impact of interventions for which reported results were of the wrong analysis.

Project Activities

The researcher first developed the theory for the new correction strategy. He will then create user-friendly code in R and in Stata for calculating the estimates and an Excel spreadsheet for conducting a meta-analysis with the estimates. He demonstrated the new strategy and the corresponding software by conducting a meta-analysis of publicly available datasets. By the end of the project, he released a user-friendly version of the software and was disseminating the results of the research at conferences and in peer-reviewed journals. The researcher also anticipated re-analyzing studies used in WWC Intervention Reports and preparing a paper illustrating discrepancies between the original conclusions and the conclusions resulting from the new approach.

People and institutions involved

IES program contact(s)

Allen Ruby

Associate Commissioner for Policy and Systems
NCER

Supplemental information

Previous Grant Number: National Opinion Research Center (NORC)
Previous University Affiliation: R305D160009

Questions about this project?

To answer additional questions about this project or provide feedback, please contact the program officer.

 

Tags

Data and AssessmentsMathematics

Share

Icon to link to Facebook social media siteIcon to link to X social media siteIcon to link to LinkedIn social media siteIcon to copy link value

Questions about this project?

To answer additional questions about this project or provide feedback, please contact the program officer.

 

You may also like

Zoomed in IES logo
Workshop/Training

Data Science Methods for Digital Learning Platform...

August 18, 2025
Read More
Zoomed in IES logo
Workshop/Training

Meta-Analysis Training Institute (MATI)

July 28, 2025
Read More
Zoomed in Yellow IES Logo
Workshop/Training

Bayesian Longitudinal Data Modeling in Education S...

July 21, 2025
Read More
icon-dot-govicon-https icon-quote