Using Instrumental Variables Analysis Coupled with Rigorous Multi-Site Impact Studies to Study the Causal Paths by which Educational Interventions Affect Student Outcomes

1. Under what conditions are such estimators internally valid and when might it be plausible to expect such conditions to exist?
2. Under what conditions are tests of the statistical significance of such estimators valid and when might it be plausible to expect such conditions to exist?
3. How do answers to the preceding questions depend on the particular estimation method used (e.g., two stage least squares vs. limited information maximum likelihood )?
4. What is the statistical precision or power of estimators based on the most appropriate estimation procedure for these analyses?
5. How do results from the instrumental variables analyses examined differ from those produced by corresponding ordinary least squares regression analysis?

Empirical parts of the proposed research used data from several large-scale, multi-site impact studies (including the Reading First Impact Study) to explore the feasibility of meeting the conditions necessary for the proposed approach. The team conducted a complete instrumental variables analysis of relationships between at least one focal mediator and at least one student outcome. This analysis started with a simple graphical and statistical analysis for a single mediator and instrument, followed by a graphical and statistical analysis for a single mediator and alternative configurations of multiple instruments, and concluded by a series of analyses involving multiple mediators and multiple instruments.

Auxiliary analyses tested whether the conditions required for a valid and reliable analysis are likely to hold for each data set considered. These analyses included: (1) assessments of the assumption that there is heterogeneity across sites in the treatment effect on mediators and whether this heterogeneity is correlated with the effect of the mediators on outcomes; (2) tests of the strength of the instruments involved and examinations of the implications of these results for the likely validity of point estimates and hypotheses tests based on these instruments and mediators; (3) tests for a range of multiple mediator models with different configurations of instruments to examine the sensitivity of these models to variation in specification and to explore the issue of multi-collinearity among instruments and predicted mediators and how this affects the results; (4) assessments of the sensitivity of estimates and inferences given possible unobserved mediators; and (5) comparisons of the performances of various instrumental variables estimation procedures such as two-stage-least-squares, limited information maximum likelihood, and random-effects quasi-maximum likelihood. These analyses also explored how the clustering of observations (e.g., students within classrooms with schools) affect the statistical properties of these estimators; examined how the relationship between site-level impacts of the intervention on mediators and their corresponding impacts of the mediator on student outcomes vary; and provided information on the implications of instrumental variable analysis on research design, especially with regard to statistical power.