Exploring Heterogeneity in Mathematics Intervention Effects Using Meta-Analysis
Co-Principal Investigator: James Lindsay
Purpose: The purpose of this project is to better understand the conditions and contexts for why one study may find that a mathematics intervention works while another study on the same intervention may not find an effect. Unfortunately, this variation or heterogeneity in intervention effects are often not explored together to determine why the intervention worked in one context but not the other. By examining the heterogeneity in treatment effects, researchers and practitioners can have a better understanding of the conditions and contexts in which a study's findings generalize, and it can improve the design of experiments in mathematics education.
Project Activities: There are two phases in this project. The first phase involves a systematic literature review of mathematics intervention studies. The second phase involves using meta-analytic methods to explore sources of intervention effect heterogeneity, grouping studies by intervention type and outcome domain.
Products: The findings from this project will provide information to practitioners, policymakers, applied researchers, and methodologists about the conditions under which treatment effects vary for mathematics interventions. In addition, the study will describe the effects of different types of mathematics interventions and outcomes, providing a platform to compare the relative potential advantages and disadvantages of different types and combinations of mathematics interventions. The researchers will also produce peer-reviewed publications describing their findings.
Setting: The researchers will include findings from mathematics interventions from settings and locations across the United States.
Sample: For the meta-analysis, only studies of mathematics interventions designed with the specific goal of improving student performance in mathematics from prekindergarten through Grade 12 will be included. The studies included in the meta-analysis will also be limited to those written in English and written no earlier than 1991.
Intervention: Not applicable for this study.
Research Design and Methods: The current study uses meta-analysis to explore sources of variation or heterogeneity in mathematics intervention effects. Without information on effect size heterogeneity, researchers and practitioners are unable to understand the circumstances (e.g., settings, intervention types, student subgroups) in which mathematics interventions are most likely to produce positive impacts.
During the first phase of the project, the researchers will conduct a systematic review of mathematics intervention studies and estimate mean treatment effects for each intervention type and outcome domain separately using meta-analysis. The process begins with a thorough search of the published and unpublished literature, followed by preliminary and advanced screening of studies for inclusion, and coding effect size estimates and study features.
The second phase of the project involves exploring within-study and between-study sources of heterogeneity using Cronbach's units, treatments, outcomes, and settings (UTOS) framework for generalizability. The researchers will collect information on the characteristics of the students in the samples, the interventions and intervention features that were implemented, the outcomes that were measured, and the characteristics of the study settings. The researchers will use meta-analytic methods to combine effect sizes from the mathematics intervention studies that meet the inclusion criteria and explore sources of heterogeneity.
Data Analytic Strategy: For studies that meet the inclusion criteria, the researchers will extract information to calculate both unconditional and conditional intervention effects (separately for each outcome type). Unconditional effects provide an overall (or average) estimate of the strength of an intervention on a given outcome. Conditional effects are estimated by controlling (or accounting) for various characteristics (or covariates) related to participants, the intervention, the outcome, and the setting. After the effect sizes are calculated, the researchers will use a series of random and mixed-effects meta-analytic models to combine effect size estimates and explore variation in the effect size distributions.