Grant Open

Practical Tools for Large-Scale Evaluation of Text Data in Randomized Trials in Education

NCER

Program: Statistical and Research Methodology in Education

Program topic(s): Core

Award amount: $894,352

Principal investigator: Luke Miratrix

Awardee:

Harvard University

Year: 2022

Project type:

Methodological Innovation

Award number: R305D220032

Purpose

The purpose of this grant is to provide machine learning tools to improve the statistical power of an education impact analysis that relies on human-scoring of text passages written by students. Machine learning can be particularly useful in studies that have more text data than time, money, and/or coders to score all of the text data by human raters. By using the text and scores provided by the available human raters to train a machine learning algorithm, computer-scoring would then be used to score the remaining text passages. This would both increase the statistical power of text-based studies and leverage key aspects of machine learning to run additional analyses for investigating potential text pattern differences between the groups.

Project Activities

The research team will first develop new models for automated scoring that align with human judgment. The team will write these models into statistical software and then tested on real text datasets. The researchers will then expand the models and software to allow the machine learning algorithm to evaluate potential differences between experimental or quasi-experimental groups in the texts produced by the groups, not just average score differences between the groups. They will continue software testing on additional real datasets to gauge its performance at discerning text pattern differences.

People and institutions involved

IES program contact(s)

Charles Laurin

Education Research Analyst

NCER

Products and publications

Products: When the software is ready, the research team will create a free, web-based, user-friendly version for use by applied researchers. The team will then create instructional materials for use in workshops and short courses for teaching users how to use the software. The team will also produce manuscripts for publication in methodological and applied peer-reviewed journals.

Publications:

Mozer, R., & Miratrix, L. (2023). Decreasing the human coding burden in randomized trials with text-based outcomes via model-assisted impact analysis. arXiv preprint arXiv:2309.13666.

Mozer, R., Miratrix, L., Relyea, J. E., & Kim, J. S. (2023). Combining Human and Automated Scoring Methods in Experimental Assessments of Writing: A Case Study Tutorial. Journal of Educational and Behavioral Statistics, 10769986231207886.