Purpose: The purpose of this project was to improve the yield from cause-probing research in education. Considerable professional consensus already exists regarding which quasi-experimental designs represent acceptable alternatives when an experiment is not feasible; these include regression discontinuity, case matching with emphasis on propensity scores, short interrupted time series, and pattern matching. This project focused on improving research using these four design or analytic principles. Although these are the strongest quasi-experimental designs currently available, the research team investigated specific ways to improve them.

Project Activities: For regression discontinuity, the research team examined different methods for improving the generalization of local average treatment effects and for handling the misallocation of treatments. For case matching, they examined the conditions under which propensity score matches produce unbiased causal inference, and the importance of local, focal matches for constructing comparison groups. For short interrupted time series, the team empirically assessed how short is "too short," and what to do with as few as three pre-intervention time points. For pattern matching, the research team explored creative uses of the design to make it more available to education researchers.

Two principal methods were used to illustrate improved design. One is Monte Carlo simulations designed to show how variation in some parameters affects an outcome under the total set of conditions built into the simulation. The other is the reanalysis of within-study comparison datasets comparing quasi-experimental results to those from randomized experiments sharing the same treatment group and thus able to serve as a valid causal benchmark for estimating how much bias reduction is achieved by one way of improving a design versus another.

Products and Publications

Book chapter

Hallberg, K., Wing, C., Wong, V.C., and Cook, T.D. (2013). Experimental Design for Causal Inference: Clinical Trial and Regression-Discontinuity Designs. In T. Little (Ed.), The Oxford Handbook of Quantitative Methods, Volume 1: Foundations (pp. 223–236). New York: Oxford University Press.

Shadish, W., and Sullivan, K. (2012). Theories of Causation in Psychological Science. In H. Cooper, P. Camic, D. Long, A. Panter, D. Rindskopf, and K.J. Sher (Eds.), APA Handbook of Research Methods in Psychology, Volume 1: Foundations, Planning, Methods, and Psychometrics (pp. 23–52). Washington, DC: American Psychological Association.

Steiner, P.M., and Cook, D.L. (2013). Matching and Propensity Scores. In T.D. Little (Ed.), The Oxford Handbook of Quantitative Methods, Volume 1: Foundations (pp. 237–259). New York: Oxford University Press.

Steiner, P.M., Wroblewski, A., and Cook, T.D. (2009). Randomized Experiments and Quasi-Experimental Designs in Educational Research. In K. Ryan, and B.J. Cousins (Eds.), The Handbook of International Education (pp. 75–95). London, UK: Sage Publications.

Book, edition specified

Wong, V.C., Wing, C., Steiner, P.M., Wong, M., and Cook, T.D. (2013). Research Designs for Program Evaluation. (2nd ed.). Hoboken, NJ: John Wiley and Sons, Inc.

Journal article, monograph, or newsletter

Cook, T.D. (2008). Waiting for Life to Arrive: A History of the Regression-Discontinuity Design in Psychology, Statistics and Economics. Journal of Econometrics, 142(2): 636–654.

Cook, T.D., and Steiner, P.M. (2009). Some Empirically Viable Alternatives to the Randomized Experiment. Journal of Policy Analysis and Management, 28(1): 165–166.

Cook, T.D., and Steiner, P.M. (2010). Case Matching and the Reduction of Selection Bias in Quasi-Experiments: The Relative Importance of Covariate Choice, Unreliable Measurement and Mode of Data Analysis. Psychological Methods, 15(1): 56–68.

Cook, T.D., and Wong, V.C. (2008). Empirical Tests of the Validity of the Regression Discontinuity Design: Implications for its Theory and its Use in Research Practice. Annales d'Economie et de Statistique, 91: 127–150.

Cook, T.D., Scriven, M., Coryn, C.L.S., and Evergreen, S.D.H. (2010). Contemporary Thinking About Causation in Evaluation: A Dialogue With Tom Cook and Michael Scriven. American Journal of Evaluation, 31(1): 105–117.

Cook, T.D., Steiner, P.M., and Pohl, S. (2009). How Bias Reduction Is Affected by Covariate Choice, Unreliability, and Mode of Data Analysis: Results From Two Types of Within-Study Comparisons. Multivariate Behavioral Research, 44(6): 828–847.

Greenwood, C.R., Bradfield, T., Kaminski, R., Linas, M., Carta, J., and Nylander, D. (2011). The Response to Intervention (RTI) Approach in Early Childhood. Focus on Exceptional Children, 43(9): 1–22.

Pohl, S., Steiner, P.M., Eisermann, J., Soellner, R., and Cook, T.D. (2009). Unbiased Causal Inference From an Observational Study: Results of a Within-Study Comparison. Educational Evaluation and Policy Analysis, 31(4): 463–479.

Shadish, W.J., and Cook, T.D. (2009). The Renaissance of Experiments. Annual Review of Psychology, 60: 607–629.

Shadish, W.R., and Steiner, P.M. (2010). A Primer on Propensity Score Analysis. Newborn and Infant Nursing Reviews, 10: 19–26.

Shadish, W.R., Galindo, R., Wong, V.C., Steiner, P.M., and Cook, T.D. (2011). A Randomized Experiment Comparing Random to Cutoff-Based Assignment. Psychological Methods, 16(2): 179–191.

Steiner, P.M., Cook, T.D., and Shadish, W.R. (2011). On the Importance of Reliable Covariate Measurement in Selection Bias Adjustments Using Propensity Scores. Journal of Educational and Behavioral Statistics, 36(2): 213–236.

Steiner, P.M., Cook, T.D., Shadish, W.R., and Clark M.H. (2010). The Importance of Covariate Selection in Controlling for Selection Bias in Observational Studies. Psychological Methods, 15(3): 250–267.

Wong, V.C., Cook, T.D., Barnett, S.W., and Jung, K. (2008). An Effectiveness-Based Evaluation of Five State Pre-Kindergarten Programs. Journal of Policy Analysis and Management, 27(1): 122–154.

Wong, V.C., Steiner, P.M., and Cook, T.D. (2013). Analyzing Regression-Discontinuity Designs With Multiple Assignment Variables: A Comparative Study of Four Estimation Methods. Journal of Educational and Behavioral Statistics, 38(2): 107–141.

** This project was submitted to and funded as an Unsolicited application in FY 2007.