Project Activities
The project team will develop methods that will render the treated and control groups comparable in terms of observed data. The team will apply weights to students' measured outcomes before comparing the groups. A widely used method of statistical adjustment in education research has been to use propensity score weights estimated from a logistic regression model. This existing method is prone to bias and instability, particularly when treated and untreated students' pre-intervention data are so different that their covariate distributions do not overlap. The project team will begin by developing the theory of the new balancing weights estimator and will implement it in software before evaluating it with simulated data. When the estimator is validated in simulations, the team will evaluate it in an empirical data set.
Structured Abstract
Research design and methods
The project team will first develop the theory of the balancing weight estimator, which can be viewed as a form of model-free random effects. Once the theory of the estimator is developed, the project team will develop methods to correctly specify the model structure for balancing weights. The team will design the proposed methods to detect nonlinearity, omitted interactions, and balance higher moments for continuous covariates. Balancing weight methods rely on hyperparameters to control the bias-variance tradeoff. The project team will develop data-driven methods for hyperparameter selection. Once the theory and methods are developed, the project team will develop a complete workflow for a balancing weights analysis. This workflow will specify all of the analytic steps necessary to apply balancing weights in an observational study. The project team will outline best practice for diagnostics, estimation of the weights, treatment effect estimation, and sensitivity analysis. The team will validate the novel methods by conducting simulation studies and a within-study comparison (evaluating the bias and variance of the balancing weights estimates in a randomized controlled trial where statistical adjustment is truly unnecessary).
User Testing: The project team will produce software and software manuals as well as guidance for using the novel balancing weight methods. The team will test these products by using them in courses that they teach and at workshops that they plan to give at education research conferences. The team will iterate on their products in response to feedback from class and workshop participants.
Use in Applied Education Research: The project team will apply their novel balancing weights methods in a quasi-experimental analysis of the effectiveness of voluntary pre-K services in the Wake County Public School System on improving math and reading outcomes at the end of grade 3.
People and institutions involved
IES program contact(s)
Products and publications
Products: The project team will use the results of simulations and theoretical derivations to develop clear study guidelines and reporting standards for their novel balancing weights methods. The team will also disseminate these research findings through seminars, short courses, conference presentations, and peer-reviewed journal manuscripts. The team will also develop easy-to-use software in the R language so that researchers can implement these best practices.
ERIC Citations: Find available citations in ERIC for this award here.
Related projects
Supplemental information
Co-Principal Investigators: Ben-Michael, Eli; Feller, Avi
Questions about this project?
To answer additional questions about this project or provide feedback, please contact the program officer.
