Co-Principal Investigator: David Rindskopf

Purpose: The purpose of this project is to develop meta-analytic methods for synthesizing one kind of nonrandomized experiment, the single-subject design. These designs are widely used in educational research settings (Kazdin, 2003; Kratochwill & Levin, 1992), and are among the strongest nonrandomized experiments (Shadish, Cook & Campbell, 2002). Although meta-analysis has been applied comparatively rarely to these designs, sufficient literature exists to believe that the time is propitious for further development of this approach.

Project Activities: The project will address the following research goals:

The first goal of this project is to review the literature on meta-analysis of single-subject designs. The review will have two purposes. One is to compile a bibliography of meta-analyses of single subject designs in education. The other is to identify a comprehensive set of methods for doing meta-analysis of single-subject designs, along with the strengths and weaknesses of each method.

The second goal is to further develop multilevel statistical models for the analysis and synthesis of these designs. Such models will prove to have the most desirable properties for this task.

The third goal is to conduct several meta-analyses of single-subject data sets. Such meta-analyses will (a) illustrate the multilevel modeling techniques that are developed as part of the second goal, (b) serve as a concrete opportunity to review and critique criteria for including or excluding single-subject designs from meta-analysis, (c) serve as an opportunity to develop one or more prototype coding manuals that can be used more widely by educational researchers to code individual single-subject designs that are being included in meta-analysis, and (d) permit the development of a sample script for software that can do the multilevel analyses to be developed.

The fourth goal is to produce a book on methods for meta-analysis of single-subject designs that will serve to disseminate all the products developed under the first three goals.

Products: The expected products from this study include:

1. An annotated bibliography of meta-analyses of single-subject designs in education,
2. An annotated bibliography of methods for doing meta-analyses of single-subject designs,
3. A manuscript reviewing the literature on the meta-analysis of single-subject designs, outlining the major methods for doing such work, and fully describing the strengths and weaknesses of the existing methods,
4. A set of single-subject data sets on which current and future methods can be tested,
5. A manuscript on the use of multilevel models to conduct meta-analyses of single subject designs,
6. At least two meta-analyses of single-subject data sets in education,
7. A manuscript discussing inclusion and exclusion criteria for use in meta-analyses of single-subject designs,
8. One or more sample coding manuals for doing meta-analyses of single-subject designs,
9. Sample scripts for software designed for meta-analyses of single-subject designs, including at least one commercial program and one freeware program, and
10. A book disseminating the results of the proposed research.

Research Design and Methods: The project will (a) conduct a review of the literature on meta-analyses of single-subject designs for two purposes: to compile a bibliography of meta-analyses of single-subject designs in education and to identify a comprehensive set of methods for doing meta-analysis of single-subject designs, (b) further develop multilevel statistical models for the analysis and synthesis of these designs, (c) conduct several meta-analyses of single-subject data sets, which will serve multiple purposes (e.g., illustrate the multilevel modeling techniques, develop prototype coding manuals, develop sample script for software on multilevel analyses), and (d) produce a book on methods for meta analyses of single-subject designs that will serve to disseminate all the products developed.

Data Analytic Strategy: For the literature review, the researchers will use standard literature review techniques to search literatures in education, psychology, statistics, and medicine, and will include any studies of single-subject designs (e.g., ABAB, ABAB reversal designs). Development of multilevel models will be conducted through comparisons of results on real data sets using Bayesian methods versus popular multilevel modeling programs (e.g., HLM and MLWin). Furthermore, the researchers will investigate whether nonlinear models are more appropriate for some single-subject data. The meta-analyses the researchers plan to conduct will illustrate the multilevel modeling techniques they develop, as well as allow the researchers to develop and recommend criteria for including or excluding single-subject designs from meta-analyses. A detailed sample script will then be developed to assist educational researchers who are not familiar with writing script for the many kinds of computer programs that implement the multilevel models used in this study. In addition, a script for one commercial package (e.g, HLM, MLwiN) and one freeware package (e.g., WinBUGS, MIX for Windows) will be developed.

Products and Publications

Journal article, monograph, or newsletter

Shadish, W.R., and Rindskopf, D.M. (2007). Methods for Evidence-Based Practice: Quantitative Synthesis of Single-Subject Designs. New Directions for Evaluation, 113: 95–109. doi:10.1002/ev.217

Shadish, W.R., Brasil, I.C.C., Illingworth, D.A., White, K., Galindo, R., Nagler, E.D., and Rindskopf, D.M. (2009). Using UnGraph to Extract Data From Image Files: Verification of Reliability and Validity. Behavior Research Methods, 41(1): 177–183. doi:10.3758/BRM.41.1.177

Shadish, W.R., Rindskopf, D.M., and Hedges, L.V. (2008). The State of the Science in the Meta Analysis of Single-Case Experimental Designs. Evidence-Based Communication Assessment and Intervention, 2(3): 188–196.

* This has an H324 code but has been funded by NCSER.