Co-Principal Investigators: Citkowicz, Martyn; Lindsay, James

Purpose: The purpose of this project was to better understand the conditions and contexts for why one study may find that a mathematics intervention works while another study on the same intervention may not find an effect. Unfortunately, this variation or heterogeneity in intervention effects are often not explored together to determine why the intervention worked in one context but not the other. By examining the heterogeneity in treatment effects, researchers and practitioners can have a better understanding of the conditions and contexts in which a study's findings generalize, and it can improve the design of experiments in mathematics education.

Project Activities: The project included two phases. The first phase included a systematic literature review of randomized experiments of mathematics intervention studies from 1991 to 2017. The second phase utilized meta-analytic methods to explore sources of intervention effect heterogeneity by examining variation in effects by sample, intervention, outcome, setting characteristics.

Key Outcomes: Information about key outcomes and study findings will be reported when peer reviewed publications are available.

Structured Abstract

Setting: The researchers included findings from mathematics interventions across all settings, including urban, suburban, and rural and across all geographical regions of the United States, from the southwest to the northeast.

Sample: The sample of studies included in the meta-analysis consisted of 190 randomized experiments of interventions designed to improve mathematics learning for students in prekindergarten through Grade 12 and published between January 1991 and August 2017. The studies produced 1,110 effect size estimates that represent more than a quarter of a million students, 72% of which were elementary school students. The sample was relatively diverse: 52% were male, 40% were White, 33% were Black, 25% were Hispanic, 57% were economically disadvantaged, 22% were English learners, and 20% received special education services.

Intervention: Not applicable for this study.

Research Design and Methods: The study used meta-analysis to explore sources of variation or heterogeneity in mathematics intervention effects. Without information on effect size heterogeneity, researchers and practitioners are unable to understand the circumstances (e.g., settings, intervention types, student subgroups) in which mathematics interventions are most likely to produce positive impacts.

During the first phase of the project, the researchers conducted a systematic review of mathematics intervention studies and estimated mean treatment effects for each intervention type (including curriculum, instructional/pedagogical, and supplemental time) using meta-analysis. The process began with a thorough search of the published and unpublished literature, followed by preliminary and advanced screening of studies for inclusion, and coding effect size estimates and study features.

The second phase of the project involved exploring within-study and between-study sources of heterogeneity using Cronbach's units, treatments, outcomes, and settings (UTOS) framework for generalizability. The researchers collected information on the characteristics of the students in the samples, the interventions and intervention features that were implemented, the outcomes that were measured, and the characteristics of the study settings. The researchers used meta-analytic methods to combine effect sizes from the mathematics intervention studies that met the inclusion criteria and explored sources of heterogeneity.

Data Analytic Strategy: The researchers used a model building process to determine the final analytic model. They ran four separate models which each included a group of moderators corresponding to a component of Cronbach's UTOS framework. For instance, the model corresponding to the "O" component of UTOS (i.e., outcomes) included outcome moderators such as dummy codes for the outcome domain (e.g., algebra assessment), standardized or researcher-generated achievement measure, and timing (e.g., immediate or delayed posttest), in addition to methods moderators (e.g., outcome type, level of random assignment, attrition, and publication status). From these four models, the researchers selected moderators with p-values less than .10 for inclusion into a final combined model.

Products and Publications

Publicly available data

The researchers developed a database of mathematics intervention effects that consists of an interactive web application designed to allow policymakers, practitioners, decisionmakers, intervention developers, and researchers to explore the pool of mathematics intervention effects from the last 25 years to better understand the conditions where various mathematics programs work best. The database will be published on the American Institutes of Research's (AIR's) Methods of Synthesis and Integration Center (MOSAIC) website in early 2021.