Purpose: This project explored the theoretical properties and empirical implications of using instrumental variables analysis coupled with rigorous multi-site impact studies to estimate the causal paths by which educational interventions produce effects on students. To this end, it addressed five research questions:

1. Under what conditions are such estimators internally valid and when might it be plausible to expect such conditions to exist?
2. Under what conditions are tests of the statistical significance of such estimators valid and when might it be plausible to expect such conditions to exist?
3. How do answers to the preceding questions depend on the particular estimation method used (e.g., two stage least squares vs. limited information maximum likelihood )?
4. What is the statistical precision or power of estimators based on the most appropriate estimation procedure for these analyses?
5. How do results from the instrumental variables analyses examined differ from those produced by corresponding ordinary least squares regression analysis?

Project Activities: The project included a theoretical component that synthesized existing information on the topic through a survey of relevant econometrics, statistics literatures, and consultations with methodological experts in the field. Consultations with researchers who are using the proposed approach to study causal relationships in fields outside of education research was also included. The synthesis were presented in seminars and papers that are broadly accessible to education researchers.

Empirical parts of the proposed research used data from several large-scale, multi-site impact studies (including the Reading First Impact Study) to explore the feasibility of meeting the conditions necessary for the proposed approach. The team conducted a complete instrumental variables analysis of relationships between at least one focal mediator and at least one student outcome. This analysis started with a simple graphical and statistical analysis for a single mediator and instrument, followed by a graphical and statistical analysis for a single mediator and alternative configurations of multiple instruments, and concluded by a series of analyses involving multiple mediators and multiple instruments.

Auxiliary analyses tested whether the conditions required for a valid and reliable analysis are likely to hold for each data set considered. These analyses included: (1) assessments of the assumption that there is heterogeneity across sites in the treatment effect on mediators and whether this heterogeneity is correlated with the effect of the mediators on outcomes; (2) tests of the strength of the instruments involved and examinations of the implications of these results for the likely validity of point estimates and hypotheses tests based on these instruments and mediators; (3) tests for a range of multiple mediator models with different configurations of instruments to examine the sensitivity of these models to variation in specification and to explore the issue of multi-collinearity among instruments and predicted mediators and how this affects the results; (4) assessments of the sensitivity of estimates and inferences given possible unobserved mediators; and (5) comparisons of the performances of various instrumental variables estimation procedures such as two-stage-least-squares, limited information maximum likelihood, and random-effects quasi-maximum likelihood. These analyses also explored how the clustering of observations (e.g., students within classrooms with schools) affect the statistical properties of these estimators; examined how the relationship between site-level impacts of the intervention on mediators and their corresponding impacts of the mediator on student outcomes vary; and provided information on the implications of instrumental variable analysis on research design, especially with regard to statistical power.

Products and Publications

Journal article, monograph, or newsletter

Bloom, H.S. (2012). Modern Regression Discontinuity Analysis. Journal of Research on Educational Effectiveness, 5(1): 43–82.

Bloom, H.S. (2012). Comments: Statistical Analysis for Multisite Trials. Journal of Research on Educational Effectiveness, 5(3): 333–335.

Raudenbush, S.W., Reardon, S.F., and Nomi, T. (2012). Statistical Analysis for Multisite Trials Using Instrumental Variables With Random Coefficients. Journal of Research on Educational Effectiveness, 5(3): 303–332.

Reardon, S.F., and Raudenbush, S.W. (2013). Under What Assumptions do Site-By-Treatment Instruments Identify Average Causal Effects?. Sociological Methods & Research, 42(2), 143–163.

Reardon, S.F., Unlu, F., Zhu, P., and Bloom, H.S. (2014). Bias and Bias Correction in Multisite Instrumental Variables Analysis of Heterogeneous Mediator Effects. Journal of Educational and Behavioral Statistics, 39(1), 53–86.