Grant Open

Fully Latent Principal Stratification: A New Framework for Big, Complex Implementation Data from Education RCTs

NCER

Program: Statistical and Research Methodology in Education

Program topic(s): Core

Award amount: $891,895

Principal investigator: Tiffany Whittaker

Awardee:

University of Texas, Austin

Year: 2021

Project type:

Methodological Innovation

Award number: R305D210036

Purpose

Unlike their more traditional counterparts, computer-based interventions typically allow researchers and administrators to collect implementation data automatically in the form of log or clickstream data. Log data from technology RCTs present an unprecedented opportunity for researchers to use the fine-grained and rich data to help understand how, why, and when online interventions work. Log data, however, present a challenge, in that they differ in structure and size from data commonly encountered in studies of causal mechanisms. Computer log data are highly multivariate, multilevel, and messy. Statistical methods developed for computer log data do not adapt easily to causal models. The purpose of this grant is to develop fully latent principal stratification (FLPS), a data science tool for modeling large complex multivariate implementation data from randomized controlled trials (RCTs) in education. FLPS combines principal stratification and structural equation modeling techniques to examine treatment effect heterogeneity as a function of implementation constructs measured by longitudinal, multidimensional, and mixed-type data, such as computer log data from educational technology applications.

Project Activities

The researchers will formulate FLPS models for a variety of measurement scenarios and study their feasibility, estimation properties, robustness, and associated model-checking techniques. They will also develop and disseminate a set of vignettes with worked examples and open-source code in R that will make it possible for practitioners to fit and make inferences from multivariate intermediate data using FLPS models.

Products and publications

Products: The research team will publish in peer-reviewed journals, present at conferences, and provide workshops at conferences and universities on FLPS models and the software for estimating them.

ERIC Citations: Find available citations in ERIC for this award here.

Gagnon-Bartsch, J. A., Sales, A. C., Wu, E., Botelho, A. F., Erickson, J. A., Miratrix, L. W., & Heffernan, N. T. (2023). Precise unbiased estimation in randomized experiments using auxiliary observational data. Journal of Causal Inference, 11(1), 20220011.

Kang, H. A., Sales, A., & Whittaker, T. A. (2024). Flow with an intelligent tutor: A latent variable modeling approach to tracking flow during artificial tutoring. Behavior Research Methods, 56(2), 615-638.

Lee, S., Adam, S., Kang, H. A., & Whittaker, T. A. (2022, July). Fully Latent Principal Stratification: combining PS with model-based measurement models. In The Annual Meeting of the Psychometric Society (pp. 287-298). Cham: Springer Nature Switzerland.

Lee, S., Sales, A. C., Kang, H. A., & Whittaker, T. A. (2023). Fully Latent Principal Stratification With Measurement Models. arXiv preprint arXiv:2309.04047.

Vanacore, K., Gurung, A., Sales, A., & Heffernan, N. T. (2024, March). The Effect of Assistance on Gamers: Assessing The Impact of On-Demand Hints & Feedback Availability on Learning for Students Who Game the System. In Proceedings of the 14th Learning Analytics and Knowledge Conference (pp. 462-472).

Vanacore, K., Ottmar, E., Liu, A., & Sales, A. (2024). Remote monitoring of implementation fidelity using log-file data from multiple online learning platforms. Journal of Research on Technology in Education, 1-21.

Vanacore, K., Sales, A., Liu, A., & Ottmar, E. (2023, July). Benefit of gamification for persistent learners: Propensity to replay problems moderates algebra-game effectiveness. In Proceedings of the Tenth ACM Conference on Learning@ Scale (pp. 164-173).

Supplemental information

Co-Principal Investigators: Kang, Hyeon-Ah; Sales, Adam

Questions about this project?

To answer additional questions about this project or provide feedback, please contact the program officer.