Grant Open

Consequences of Selective Reporting Bias in Education Research

NCER

Program: Statistical and Research Methodology in Education

Program topic(s): Core

Award amount: $896,931

Principal investigator: Martyna Citkowicz

Awardee:

American Institutes for Research (AIR)

Year: 2022

Award period: 3 years 6 months (09/01/2022 - 02/28/2026)

Project type:

Methodological Innovation

Award number: R305D220026

Purpose

The most popular meta-analytic methods have serious limitations in diagnosing and adjusting for selective reporting, especially when there are dependencies among multiple effects from primary studies, a widespread occurrence. For meta-analysis of independent effects, selection models have shown promise in flexibly capturing complex reporting patterns while providing adjusted meta-analytic estimates, but no existing model also simultaneously addresses effect size dependencies. The purpose of this project is to develop two selection models that will simultaneously account for selective reporting and effect size dependencies.

Project Activities

This model is based on the beta-density distribution and the other model is based on theoretically important p-value cut points. After these models are derived, their performance will be tested via Monte Carlo simulation studies under varying conditions, such as different sample sizes and different degrees of selection bias.

People and institutions involved

IES program contact(s)

Charles Laurin

Project contributors

Joshua Polanin

Co-principal investigator

James Pustejovsky

Co-principal investigator

Ryan Williams

Co-principal investigator

Products and publications

The research team will then develop an R software package and a Shiny app for meta-analysis researchers to apply the new models to their own research. In order to help researchers use the new software, the project team will prepare several tutorials, prepare a webinar, and hold in-person trainings. They will also conduct and publish the results of a second-order meta-analysis.