Skip to main content

Breadcrumb

Home arrow_forward_ios Information on IES-Funded Research arrow_forward_ios Exploring Heterogeneity in Mathemat ...
Home arrow_forward_ios ... arrow_forward_ios Exploring Heterogeneity in Mathemat ...
Information on IES-Funded Research
Grant Closed

Exploring Heterogeneity in Mathematics Intervention Effects Using Meta-Analysis

NCER
Program: Education Research Grants
Program topic(s): Science, Technology, Engineering, and Mathematics (STEM) Education
Award amount: $599,104
Principal investigator: Ryan Williams
Awardee:
American Institutes for Research (AIR)
Year: 2017
Project type:
Exploration
Award number: R305A170146

Purpose

The purpose of this project was to better understand the conditions and contexts for why one study may find that a mathematics intervention works while another study on the same intervention may not find an effect. Unfortunately, this variation or heterogeneity in intervention effects are often not explored together to determine why the intervention worked in one context but not the other. By examining the heterogeneity in treatment effects, researchers and practitioners can have a better understanding of the conditions and contexts in which a study's findings generalize, and it can improve the design of experiments in mathematics education.

Project Activities

The project included two phases. The first phase included a systematic literature review of randomized experiments of mathematics intervention studies from 1991 to 2017. The second phase utilized meta-analytic methods to explore sources of intervention effect heterogeneity by examining variation in effects by sample, intervention, outcome, setting characteristics.

Structured Abstract

Setting

The researchers included findings from mathematics interventions across all settings, including urban, suburban, and rural and across all geographical regions of the United States, from the southwest to the northeast.

Sample

The sample of studies included in the meta-analysis consisted of 190 randomized experiments of interventions designed to improve mathematics learning for students in prekindergarten through Grade 12 and published between January 1991 and August 2017. The studies produced 1,110 effect size estimates that represent more than a quarter of a million students, 72% of which were elementary school students. The sample was relatively diverse: 52% were male, 40% were White, 33% were Black, 25% were Hispanic, 57% were economically disadvantaged, 22% were English learners, and 20% received special education services.
Intervention
Not applicable for this study.

Research design and methods

The study used meta-analysis to explore sources of variation or heterogeneity in mathematics intervention effects. Without information on effect size heterogeneity, researchers and practitioners are unable to understand the circumstances (e.g., settings, intervention types, student subgroups) in which mathematics interventions are most likely to produce positive impacts.

Data analytic strategy

The researchers used a model building process to determine the final analytic model. They ran four separate models which each included a group of moderators corresponding to a component of Cronbach's UTOS framework. For instance, the model corresponding to the "O" component of UTOS (i.e., outcomes) included outcome moderators such as dummy codes for the outcome domain (e.g., algebra assessment), standardized or researcher-generated achievement measure, and timing (e.g., immediate or delayed posttest), in addition to methods moderators (e.g., outcome type, level of random assignment, attrition, and publication status). From these four models, the researchers selected moderators with p-values less than .10 for inclusion into a final combined model.

Key outcomes

Information about key outcomes and study findings will be reported when peer reviewed publications are available.

People and institutions involved

IES program contact(s)

Christina Chhin

Education Research Analyst
NCER

Products and publications

Publicly available data

The researchers developed a database of mathematics intervention effects that consists of an interactive web application designed to allow policymakers, practitioners, decisionmakers, intervention developers, and researchers to explore the pool of mathematics intervention effects from the last 25 years to better understand the conditions where various mathematics programs work best. The database will be published on the American Institutes of Research's (AIR's) Methods of Synthesis and Integration Center (MOSAIC) website in early 2021.

Supplemental information

Co-Principal Investigators: Citkowicz, Martyn; Lindsay, James

During the first phase of the project, the researchers conducted a systematic review of mathematics intervention studies and estimated mean treatment effects for each intervention type (including curriculum, instructional/pedagogical, and supplemental time) using meta-analysis. The process began with a thorough search of the published and unpublished literature, followed by preliminary and advanced screening of studies for inclusion, and coding effect size estimates and study features.

The second phase of the project involved exploring within-study and between-study sources of heterogeneity using Cronbach's units, treatments, outcomes, and settings (UTOS) framework for generalizability. The researchers collected information on the characteristics of the students in the samples, the interventions and intervention features that were implemented, the outcomes that were measured, and the characteristics of the study settings. The researchers used meta-analytic methods to combine effect sizes from the mathematics intervention studies that met the inclusion criteria and explored sources of heterogeneity.

Questions about this project?

To answer additional questions about this project or provide feedback, please contact the program officer.

 

Tags

Mathematics

Share

Icon to link to Facebook social media siteIcon to link to X social media siteIcon to link to LinkedIn social media siteIcon to copy link value

Questions about this project?

To answer additional questions about this project or provide feedback, please contact the program officer.

 

You may also like

Blue 3 Placeholder Pattern 1
Statistical Analysis Report

2024 NAEP Mathematics Assessment: Results at Grade...

Author(s): National Center for Education Statistics (NCES)
Publication number: NCES 2024217
Read More
Zoomed in IES logo
Statistics in Brief

NAEP Mathematics 2024 State and District Snapshot ...

Author(s): National Center for Education Statistics (NCES)
Publication number: NCES 2024219
Read More
Zoomed in IES logo
First Look / ED TAB

TIMSS 2023 U.S. Highlights Web Report

Author(s): Catharine Warner-Griffin, Grace Handley, Benjamin Dalton, Debbie Herget
Publication number: NCES 2024184
Read More
icon-dot-govicon-https icon-quote