Skip Navigation
Funding Opportunities | Search Funded Research Grants and Contracts

IES Grant

Title: Bayesian Analysis of Academic Outcomes from Single-Case Experimental Designs
Center: NCER Year: 2019
Principal Investigator: Van Norman, Ethan Awardee: Lehigh University
Program: Statistical and Research Methodology in Education      [Program Details]
Award Period: 3 Years (07/01/19–06/30/22) Award Amount: $899,769
Type: Methodological Innovation Award Number: R305D190023
Description:

Co-Principal Investigator: Klingbeil, David; Pustejovsky, James

Purpose: Single-Case Experiment Designs (SCEDs) are a flexible methodology in which applied education researchers and practitioners can evaluate the effectiveness of academic interventions with students that have severe learning needs. Replication of functional relations within and across participants, and across studies is necessary to establish evidence-based practices using SCEDs. However, research regarding how outcomes from SCEDs should be summarized within and across studies to identify evidence-based practices is ongoing. The purpose of this project is to develop and validate a free website that applied education researchers and practitioners may use to estimate between-case effect sizes (ESs) within a Bayesian framework. Bayesian analysis has shown promise in estimating treatment effects when the number of observations and subjects within a study are limited, which is common in SCEDs. Current recommended SCED ESs rely upon frequentist approaches and large sample theory and solutions are not always tractable when complex data patterns are modeled. Further, the statistical skills required to estimate those ESs is often beyond the skill set of applied education researchers. The free-to-use website calculator will use a point and click interface and will not require users to download software or modify syntax to analyze SCED outcomes.

Project Activities: The team will develop and evaluate the performance of a Bayesian framework to estimate two types of SCED ESs: a multilevel between subjects metric and a generalized least squares metric developed. Both effect sizes are theoretically comparable to effect sizes used in between-group research but have demonstrated questionable technical properties when estimated within a frequentist framework in the presence of complex data patterns. They will use a combination of extant analysis and Monte Carlo simulation to develop and evaluate the Bayesian estimator. During the first stage of the project they will conduct a systematic literature review to identify and extract data from published and unpublished SCED studies that evaluated academic interventions. All extracted data will be made publicly available. They will then conduct descriptive and inferential analyses for each academic skill area to identify common conditions observed in SCED studies (e.g., number of participants, typical treatment effects, etc.) to inform subsequent simulations. During the second stage of the project researchers will conduct simulations to generate hypothetical SCED data and will use that data to compare the performance of SCED ESs estimated using frequentist approaches, Bayesian analysis using uniformed priors, and Bayesian analysis using informed priors. During the third stage the team will develop the Bayesian effect size calculator website and pilot its use with applied education researchers. Based upon quantitative and qualitative feedback we collect from those researchers, we will refine the website prior to its formal launch.

Publications and Products

Chen, M., Pustejovsky, J. E., Klingbeil, D. A., & Van Norman, E. R. (2023). Between-case standardized mean differences: Flexible methods for single-case designs. Journal of School Psychology, 98, 16–38.

Van Norman, E. R., Boorse, J., & Klingbeil, D. A. (2022). The relationship between visual depictions of rate of improvement and quantitative effect sizes in academic single-case experimental design studies. Journal of Behavioral Education, 1–22.


Back