Grant Closed

Investigating the Potential of Machine Learning Methods for Identifying Impact Variation in Randomized Control Trials

NCER

Program: Statistical and Research Methodology in Education

Program topic(s): Core

Award amount: $899,995

Principal investigator: Pei Zhu

Awardee:

MDRC

Year: 2022

Award period: 3 years (07/01/2022 - 06/30/2025)

Project type:

Methodological Innovation

Award number: R305D220028

Purpose

In research on the effects of education interventions, there is interest not only in the overall average treatment effect but also in effects for subgroups of sample members and whether those effects differ. The purpose of this grant is to investigate the value and limitations of using machine learning to detect the presence of heterogeneous subgroup impacts in randomized control trials (RCTs) in education and other policy domains. The machine learning approach does not require researchers to pre-specify subgroups, as is typically important when using standard multiple comparison procedures, and machine learning can be used when there are a large number of candidate characteristics to examine.

Project Activities

The research team will use machine learning to conduct a secondary analysis of data from Career Academies, Growth Mindset, and the Accelerated Studies in Associate Program at CUNY and Ohio. With the secondary analysis, the researchers will determine whether machine learning replicates the findings of the original analyses and whether it identifies additional and theoretically meaningful subgroup effects not identified by the original analyses. The team will also conduct a Monte Carlo simulation study to investigate the circumstances under which the machine learning approach would be potentially more useful in multi-site RCTs than conventional subgroup analyses.

People and institutions involved

IES program contact(s)

Charles Laurin

Products and publications

The products of the grant include user-friendly R software for conducting machine learning subgroup analysis for RCTs, an instructional webinar for using the software, a conference presentation, and two research papers.