Structured Abstract
Research design and methods
Currently, multilevel regression and poststratification (MRP) and related methods are state of the art for small-area estimation and adjustment for nonrandom samples in social science, and inverse-probability weighting is the standard design-based method. We propose to combine these approaches by jointly modeling the outcome, predictors, and log weights, an approach which should now be feasible by fitting a simultaneous-equations model using Stan.
The approach will use the following statistical methods: Bayesian inference, sampling theory, bootstrapping, Monte Carlo methods, multilevel regression, and poststratification. The core of the proposed method is a new theoretical idea for embedding sampling weights into a joint distribution with modeled outcomes, along with analytical work mapping population to sampling distribution, and a computational implementation using probabilistic programming. Validation will be performed in three ways: simulated data, artificially simulated subsamples of real data, and real data validated to external benchmarks.
User testing. Education methodologists will be able to experiment with our method and generalize it by extending our software implementation, for example by setting up their own regression models or swapping in machine-learning predictions. Our evaluations will be transparent with full code, so that methodologists can also perform their own evaluations on other datasets.
Applied education researchers will be able to try out the model with their data using scripts that we will provide, following the steps in our case studies, following the model of our multilevel regression and poststratification case study (https://bookdown.org/jl5522/MRP-case-studies/) and Stan case studies for education (e.g., https://mc-stan.org/users/documentation/case-studies/tutorial_twopl.html). The proposal is to develop a new method and also to implement it in a user-friendly and accessible way.
People and institutions involved
IES program contact(s)
Project contributors
Products and publications
Anticipated statistical and methodological products include: (1) a new method for combining regression (including multilevel modeling and small-area estimation) and inverse-probability modeling using Bayesian inference; (2) validations of this method using real and simulated data; (3) an efficient and user-friendly computational implementation in the open-source probabilistic programming language Stan, with interfaces to R, Python, Stata, Julia, and other general-purpose software; (4) journal publications deriving, explaining, and demonstrating the method; (5) inclusion of these ideas in forthcoming textbooks; (5) tutorial materials and examples specifically focused on education research, including applications to widely-used education datasets (new or improved methods, toolkit, software, guidelines, compendia, review papers, articles) the research team will develop.
Dissemination is essential to this project, as the goal is to develop methods that will be relevant to generalization for item-response models, surveys, experiments, and observational studies that arise in education research in the presence of nonrepresentative samples. As such, we plan to develop full case studies demonstrating the use of these methods, and free and open discussion regarding these tools and case studies will also be available through the online forums for Stan users and developers. We also plan to disseminate the findings through peer-reviewed journal manuscripts, conference presentations, and workshops at major conferences, as well as in forthcoming textbooks and on social media through our widely-read blog. To enable unrestricted use by the largest community of users for education and industry, all of our software is free and open-source. With the IES-supported project Stan, we have a demonstrated track record of disseminating Bayesian methods and software that have been widely adapted by education researchers.
Use in applied education research. Our method and product will be developed and used to improve internal and external validity in an ongoing study of the North Carolina pre-K program. More generally, it will be used in applied education research for analysis of non-representative data for which there are sampling weights. This describes a large number of studies, including the National Assessment of Educational Progress, the Program for International Student Assessment, the National Longitudinal Transition Study, High School and Beyond, Early Childhood Longitudinal Study, and many others. The proposed research will enable researchers who are analyzing such data to integrate weighting into small-area estimation and multilevel regression and poststratification, resulting in more reliable generalizations of inferences to populations of interest.
Related projects
Questions about this project?
To answer additional questions about this project or provide feedback, please contact the program officer.
