Weighting Methods for Mediation Analysis in Experimental and Quasi-Experimental Multilevel Data

Education researchers often seek to ascertain whether changes in student or teacher experiences mediate the treatment effects of an intervention on student or teacher outcomes. Many of the randomized evaluations funded by the Institute of Education Sciences and other agencies seek to assess the intention-to-treat (ITT) effects of an intervention on student outcomes. When the targeted intermediate experiences are explicitly theorized, the evaluation researchers often report the ITT effects on the intermediate experiences as well. Knowing the ITT effects on the intermediate experiences is essential but inadequate for determining whether changing the targeted intermediate experiences would in fact lead to a change in the student outcomes.

If an intervention does not improve the intended outcomes, then one possibility is that flaws in implementation weakened its impact on the targeted educational experiences. A second possibility is that the intervention did change the educational experiences as expected, but no change in student outcomes resulted. The theory could be flawed because hypothesized relationships between the intermediate experiences and the student outcomes do not hold or because the potential benefits for students were offset by the worsening of some other intermediate experiences not targeted by the intervention. Furthermore, even if an intervention does improve student outcomes, there is still a possibility that the intervention theory might have misrepresented the actual causal mechanisms.

The primary goal of this project is to develop a series of propensity-score-based weighting procedures for revealing the mediation mechanisms in multi-level educational settings. The researchers will use a combination of theory, simulation studies, and secondary data analyses to develop and to test the weighting methods and to compare their methods with path analysis and the instrument variable method. The researchers also plan to produce software that can be readily used by education researchers to implement the weighting procedures.

Products and Publications

Journal article, monograph, or newsletter

Hong, G., Deutsch, J., and Hill, H.D. (2015). Ratio-of-Mediator-Probability Weighting for Causal Mediation Analysis in the Presence of Treatment-by-Mediator Interaction. Journal of Educational and Behavioral Statistics, 40(3): 307–340.

Qin, X., and Hong, G. (2017). A Weighting Method for Assessing Between-Site Heterogeneity in Causal Mediation Mechanism. Journal of Educational and Behavioral Statistics, 42(3), 308–340.