Description:

Co-Principal Investigator: Beretvas, Natasha; Ferron, John

Purpose: The project team further developed the multilevel modeling approach for combining the results of studies that use single-subject experimental designs (SSED). The study empirically investigated the approach using both real data and simulation studies.

SSED research allows researchers to investigate intervention effects at the individual level and allows researchers to investigate how these individual intervention effects change over time. To enhance generalizability, researchers replicate across cases. Replications may exist within a primary study (such as through the application of a multiple-baseline design) or across studies. Furthermore, there can be direct replications or systematic variation in the replications. As more replications emerge and the evidence base accumulates, the need for statistical methods designed to synthesize SSED studies' results and to explore sources of systematic variability in SSED results by employing moderator analyses grows as well.

One approach for combining single-case data within and across studies is multilevel modeling. This can be used to provide estimates of individual treatment effects and how these effects change over time. It can also estimate the average treatment effect and how this effect changes over time. Finally, it can also be used to estimate the variability in treatment effects, the effects of moderators on the treatment effect, and the pattern of a treatment's effects over time. In addition, the models are flexible enough to handle: (a) the nesting of observations and of outcomes within cases and the nesting of cases within studies; (b) a variety of forms for the growth trajectory within each phase of the design (e.g., linear, curvilinear); (c) alternative dependent error structures for the growth trajectories (e.g., first order autoregressive, toeplitz); (d) heterogeneous variances (within cases, across cases, or across studies); (e) different types of outcomes (e.g., continuous, count); and (f) standardized or unstandardized raw data or effect size measures.

Project Activities: The research entailed several studies designed to address major complications encountered when synthesizing results from SSED research. The first set of studies was designed for synthesizing results from SSED studies with outcomes in the same metric. The initial study focused on the basic three-level model for the meta-analysis of SSED studies with the same outcome. A series of additional studies targeted complications encountered in the synthesis of SSED studies' results, including: autocorrelation; count data outcomes; designs with multiple outcomes and/or settings; models with nonlinear growth trajectories; and the use of bootstrapping estimation techniques. The second set of studies assessed synthesis of standardized raw data and effect sizes necessary for meta-analyzing results from SSED studies employing outcomes on different metrics. For each multilevel model extension being investigated, real SSED data was analyzed and a simulation study conducted testing estimation of the relevant model under realistic conditions.

Four types of dissemination products were used to share the results with the wider educational community. These included research presentations and publications, workshops, a freely available SSED modeling manual (providing programs in SAS, R, and MLwiN to estimate the multilevel model and its extensions), and information on the principal investigators' web sites.

Products and Publications

Book chapter

Rindskopf, D., and Ferron, J. (2014). Using Multilevel Models to Analyze Single-Case Design Data. In T.R. Kratochwill, and J.R. Levin (Eds.), Single-Case Intervention Research: Methodological and Data-Analysis Advances. American Psychological Association.

Journal article, monograph, or newsletter

Baek, E., Beretvas, S. N., Van den Noortgate, W., & Ferron, J. M. (2020). Brief research report: Bayesian versus REML estimations with noninformative priors in multilevel single-case data. The Journal of Experimental Education, 88(4), 698–710.

Baek, E.K., Moeyaert, M., Petit-Bois, M., Beretvas, S.N., Van Den Noortgate, W., and Ferron, J.M. (2014). The Use of Multilevel Analysis for Integrating Single-Case Experimental Design Results Within a Study and Across Studies. Neuropsychological Rehabilitation, 24(3–4), 590–606.

Baek, E.K., Petit-Bois, M., Van Den Noortgate, W., Beretvas, S.N., and Ferron, J.M. (2016). Using Visual Analysis to Evaluate and Refine Multilevel Models of Single-Case Studies. The Journal of Special Education, 50(1), 18–26.

Declercq, L., Jamshidi, L., Fernandez Castilla, B., Moeyaert, M., Beretvas, S. N., Ferron, J. M., & Van den Noortgate, W. (2022). Multilevel meta-analysis of individual participant data of single-case experimental designs: One-stage versus two-stage methods. Multivariate Behavioral Research, 57(2–3), 298–317.

Ferron, J. M., Moeyaert, M., Van Den Noortgate, W., and Beretvas, S. N. (2014). Estimating Causal Effects From Multiple-Baseline Studies: Implications for Design and Analysis. Psychological Methods, 19(4), 493.

Hembry, I., Bunuan, R., Beretvas, S. N., Ferron, J. M., and Van Den Noortgate, W. (2015). Estimation of a Nonlinear Intervention Phase Trajectory for Multiple-Baseline Design Data. The Journal of Experimental Education, 83(4), 514–546.

Moeyaert, M., Rindskopf, D., Onghena, P., and Van Den Noortgate, W. (2017). Multilevel Modeling of Single-Case Data: A Comparison of Maximum Likelihood and Bayesian Estimation. Psychological Methods, 22(4): 760–778.

Moeyaert, M., Ugille, M., Ferron, J.M., Onghena, P., Heyvaert, M., Beretvas, S.N., and Van Den Noortgate, W. (2015). Estimating Intervention Effects Across Different Types of Single-Subject Experimental Designs: Empirical Illustration. School Psychology Quarterly, 30(1), 50.

Moeyaert, M., Ugille, M., Ferron, J.M., Beretvas, S.N., and Van Den Noortgate, W. (2016). The Misspecification of the Covariance Structures in Multilevel Models for Single-Case Data: A Monte Carlo Simulation Study. The Journal of Experimental Education, 84(3), 473–509.

Moeyaert, M., Ugille, M., Ferron, J. M., Beretvas, S. N., and Van Den Noortgate, W. (2014). The Influence of the Design Matrix on Treatment Effect Estimates in the Quantitative Analyses of Single-Subject Experimental Design Research. Behavior Modification, 38(5), 665–704.

Moeyaert, M., Ugille, M., Ferron, J., Beretvas, S., and Van Den Noortgate, W. (2013). Modeling External Events in the Three-Level Analysis of Multiple-Baseline Across-Participants Designs: A Simulation Study. Behavior Research Methods, 45(2): 547–559.

Moeyaert, M., Ugille, M., Ferron, J., Beretvas, S., and Van Den Noortgate, W. (2014). Three-Level Analysis of Single-Case Experimental Data: Empirical Validation. Journal of Experimental Education, 82(1): 1–21.

Moeyaert, M., Ugille, M., Ferron, J.M., Beretvas, S.N., and Van Den Noortgate, W. (2013). The Three-Level Synthesis of Standardized Single-Subject Experimental Data: A Monte Carlo Simulation Study. Multivariate Behavioral Research, 48(5), 719–748.

Petit-Bois, M., Baek, E.K., Van Den Noortgate, W., Beretvas, S.N., and Ferron, J.M. (2016). The Consequences of Modeling Autocorrelation When Synthesizing Single-Case Studies Using a Three-Level Model. Behavior Research Methods, 48(2), 803–812.

Rindskopf, D., & Ferron, J. (2014). Using Multilevel Models To Analyze Single-Case Design Data. Single-Case Intervention Research: Methodological And Statistical Advances, 221–246.

Ugille, M., Moeyaert, M., Beretvas, N., Ferron, J., and Van den Noorgate, W. (2012). Multilevel Meta-Analysis of Single-Subject Experimental Designs: A Simulation Study. Behavior Research Methods, 44(4): 1244–1254.

Ugille, M., Moeyaert, M., Beretvas, N., Ferron, J., and Van den Noorgate, W. (2014). Bias Corrections for Standardized Effect Size Estimates Used With Single-Subject Experimental Designs. Journal of Experimental Education, 82(3): 358–374.