What, When, and for Whom? Principled Estimation of Effect Heterogeneity Across Multiple Treatments, Outcomes, and Groups

The research team will capitalize on the strengths of the existing BART framework for causal inference and extend it to include the following features: multivariate models for multiple outcomes with explicit covariance structures, targeted priors for multiple treatments and subgroups, and principled strategies for detection of effect moderation. The BART framework will naturally allow for estimation of effects using flexible nonparametric response surfaces and current overlap diagnostics will be extended to handle the additional estimands induced by this framework. This software will be embedded in the existing thinkCausal framework — a causal inference tool that scaffolds analysis and provides opportunities for users to learn about the underlying concepts. A novel pre-registration wizard will help researchers develop principled but flexible analysis strategies, keeping them honest while allowing maximal use of existing data.

User Testing: The project team will use strategies including: 1) gathering feedback from users on ease of use and 2) a randomized trial to assess whether researcher conclusions using the new tool are better than those reached with standard software.

Use in Applied Education Research: It's common for educational studies to collect data on more than one treatment or more than one outcome, or to examine moderation across several different subgroups. It's likely that studies that produce estimates on a set of research questions are presenting estimates and associated uncertainty quantifications (standard errors, p-values, or confidence intervals) that are overly confident and underestimate what could be replicated (in terms of statistical significance, magnitude, or direction) in future studies. The product of this grant would be immediately useful in a wide variety of education research settings for creating more replicable estimates of these quantities.

 The project team will also address four specific aspects of applied research:

1. Supporting the Standards for Excellence in Education Research (SEER, /seer).
   1. The project will create a "wizard" to help researchers with pre-registration.
   2. The modeling tools developed will "robustify" the estimation of heterogeneous treatment effects which is crucial to generalizability of findings.
   3. All methods will be open source and the analysis tool will automatically document analytic choices as they are made, simplifying and ensuring reproducibility.
   4. Tools will be made available in an accessible, easy-to-use and scaffolded software tool that doesn't require technical expertise and helps the user understand the underlying concepts.
2. Quasi Experimental Designs. The project framework is based on Bayesian Additive Regression Trees, a machine learning algorithm demonstrated to be a strong, often superior competitor to quasi-experimental methods (for example, propensity score matching).
3. Variability in effects. The project's modeling strategy has been shown to have strong performance in estimating variable treatment effects.
4. Data Science Tools for Education Researchers: The expanded thinkCausal software package is built in the RShiny language to allow for easy access to the sophisticated algorithms for which it acts as an overlay.