Co-Principal Investigator: Jee-Seon Kim

Matching strategies for nonequivalent control group designs with multilevel data are needed because randomized experiments cannot always be conducted. In order to reduce selection bias when analyzing observational data, matching techniques like propensity score (PS) matching or Mahalanobis distance matching have gained increased popularity during the last two decades. Despite the breadth of research on matching techniques, few methodological publications exist on matching nonequivalent groups in the context of multilevel data. Research is clearly needed in that area, given the increased complexity in estimating multilevel PS models, in matching units within and across different levels, and in estimating the treatment effect with multilevel regression models.

The first goal of the project is to determine the conditions under which within- and across-cluster matching strategies produce consistent estimates of average treatment effects. This will primarily be pursued by theoretical investigations and extensive simulation studies that cover a broad range of scenarios typical to education research. The second goal is to determine the matching strategies and analytic approaches that work best in education research practice. This will be investigated using real-data (Early Childhood Longitudinal Study – Kindergarten) within-study comparisons. In evaluating the matching strategies, the researchers will test different PS estimation methods (e.g., fixed versus random effects models) and various matching techniques, including PS matching, inverse propensity matching, PS stratification, and Mahalanobis distance matching.

The ultimate purpose of the project is to develop the theoretical background for matching designs and techniques for studies that involve observational multilevel data. Based upon this work, the team will derive guidelines for education researchers who are currently using multilevel matching techniques in their analyses. It is anticipated that these guidelines will contribute to the implementation of more and better-warranted matching designs in the future. Dissemination of these guidelines is anticipated to come in the form of peer-reviewed journal publications and a website maintained by the project researchers.

Products and Publications

Book chapter

Kim, J.S., Anderson, C.J., and Keller, B. (2013). Multilevel Analysis of Assessment Data. Handbook of International Large-Scale Assessment: Background, Technical Issues, snd Methods of Data Analysis, 389–425.

Kim, J.S., Steiner, P.M., and Lim, W.C. (2015). Mixture Modeling Methods gor Causal Inference With Multilevel Data. Advances in Multilevel Modeling for Educational Research, 335–359.

M Steiner, P. (2014). Design-and Model-Based Analysis of Propensity Score Designs. In W. Wiedermann and A. von Eye (Eds), Statistics and Causality: Methods for Applied Empirical Research (pp. 333–361). Wiley Series in Probability and Statistics. Wiley.

Journal article, monograph, or newsletter

Steiner, P.M., Cook, T.D., Li, W., and Clark, M.H. (2015). Bias Reduction in Quasi-Experiments With Little Selection Theory but Many Covariates. Journal of Research on Educational Effectiveness, 8(4): 552–576.

Steiner, P.M., and Kim, Y. (2016). The Mechanics of Omitted Variable Bias: Bias Amplification and Cancellation of Offsetting Biases. Journal of Causal Inference, 4(2).

Steiner, P.M., Kim, Y., Hall, C.E., and Su, D. (2017). Graphical Models for Quasi-Experimental Designs. Sociological Methods and Research, 46(2), 155–188.

Steiner, P.M., Park, S., and Kim, Y. (2016). Identifying Causal Estimands for Time-Varying Treatments Measured With Time-Varying (Age or Grade-Based) Instruments. Multivariate Behavioral Research, 51(6), 865–8780.

West, S. G., Cham, H., Thoemmes, F., Renneberg, B., Schulze, J., and Weiler, M. (2014). Propensity Scores as a Basis for Equating Groups: Basic Principles and Application in Clinical Treatment Outcome Research. Journal of Consulting and Clinical Psychology, 82(5), 906.

Proceedings

Hall, C.E., Steiner, P.M., and Kim, J.S. (2015). Doubly Robust Estimation of Treatment Effects from Observational Multilevel Data. In van der Ark L., Bolt D., Wang WC., Douglas J., Chow SM. (eds) Quantitative Psychology Research. Springer Proceedings in Mathematics & Statistics, vol 140. (pp. 321–340). Springer, Cham.

Keller, B., Kim, J.S., and Steiner, P.M. (2015). Neural Networks for Propensity Score Estimation: Simulation Results and Recommendations. In van der Ark L., Bolt D., Wang WC., Douglas J., Chow SM. (eds), Quantitative Psychology Research. Springer Proceedings in Mathematics & Statistics, vol 140. (pp. 279–291). Springer, Cham.

Kim, J.S., Lim, W.C., and Steiner, P.M. (2017). Causal Inference with Observational Multilevel Data: Investigating Selection and Outcome Heterogeneity. In van der Ark L., Wiberg M., Culpepper S., Douglas J., Wang WC. (Eds), Quantitative Psychology. IMPS 2016. Springer Proceedings in Mathematics & Statistics, vol 196. (pp. 287–308). Springer, Cham.

Kim, J.S., and Steiner, P.M. (2015). Multilevel Propensity Score Methods gor Estimating Causal Effects: A Latent Class Modeling Strategy. In In van der Ark L., Bolt D., Wang WC., Douglas J., Chow SM. (eds), Quantitative Psychology Research. Springer Proceedings in Mathematics & Statistics, vol 140. (pp. 293–306). Springer, Cham.

Steiner, P.M., Kim, J. S., and Thoemmes, F. (2013). Matching Strategies for Observational Multilevel Data. In JSM proceedings (pp. 5020–5032).