Bayesian Inference for Experimental and Observational Studies in Education

The project had three parts. In Part I, the researchers examined the utility of Bayesian inference for randomized experiments in education. From a Bayesian perspective, experiments are dynamic, involving decisions that often rest on prior knowledge gleaned from previous studies (e.g., meta-analyses) or subjective judgments of experts. The project reviewed a select set of randomized experiments in education and reanalyze previous studies within a Bayesian framework. Special attention was paid to the use of Bayesian informative hypotheses, which allow for tests of directional hypotheses based on prior research and can mitigate the classical concern over Type I error control. Bayesian analysis of variance falls into the class of procedures that can be used to test informative hypotheses. A focused set of simulation studies was done to provide direct comparisons between Bayesian and classical approaches to issues of design and hypothesis testing. In addition, Bayesian analysis of variance was extended to cluster randomized designs.  Existing data from a randomized field trial of a large-scale reform that provided professional development in Los Angeles and from an evaluation of the Success-for-All intervention were used to supplement the simulation studies.

In Part II, the researchers examined a Bayesian approach to quasi-experimental/observational studies. The researchers focused particularly on the use of the propensity score adjustment procedures for addressing nonequivalence in such studies. Classical approaches to propensity score adjustment cannot account for uncertainties in model parameters or model choice, either of which can affect causal inferences after propensity score adjustment. The project compared the Bayesian propensity score adjustment with varying prior probabilities on model parameters that reflect reasonable choices found in education settings. This included situations in which little or no prior information is available. Specifically, a Bayesian approach to propensity score stratification, weighting, and optimal matching was developed. The researchers also examined Bayesian model averaging as a means of drawing strength from various realistic models for the propensity score. Analysis of data from the Early Childhood Longitudinal Study was used to supplement simulation studies.

Part III extended recent developments in Bayesian growth mixture modeling to the general growth mixture modeling case. In particular, researchers examined piecewise growth mixture modeling. They then examined the properties of the Bayesian piecewise growth mixture model under a variety of realistic conditions through simulation studies. Next, the Bayesian piecewise growth mixture modeling was applied to samples of data drawn from the World-class Instructional Design and Assessment Consortium that addresses response to intervention for English language learners.