Reducing Bias and Improving Efficiency of Estimated Teacher Effects from Value-Added Models

Co-Principal Investigator: McCaffrey, Daniel

The project's second component is to explore a novel approach to VAM that uses propensity scores to adjust for pre-existing differences among students taught by different teachers. This method has been successful in other disciplines to estimate causal effects from observational data but has not been explored in VAM. By relaxing the strict reliance on parametric regression models, propensity score approaches to VAM have the potential to mitigate all three sources of bias. To this end, the project will: (1) use theoretical and empirical information to build flexible models for the probabilities of individual students being in a given teacher's class; (2) explore the balance on observable characteristics these procedures provide for teachers; (3) estimate individual teacher effects using a variety of approaches, including propensity score weighted means, different forms of regression adjustment, and doubly robust methods; and 4) compare estimated teacher effects and standard errors obtained from these approaches to other estimation approaches, exploring sources of any substantial differences to understand potential shortcomings of the different approaches (e.g., regression model misspecification) including when the propensity score methods suggest classrooms cannot be made comparable.

The project's third component will addresses the other known limitation of VAM estimates of teacher effects: low precision due to small samples of students taught by each teacher and to the extensive adjustments required to control for potential biases arising from differences among classrooms in their students' background variables and prior achievement. The project will adapt methods of small-area and shrinkage estimation to develop methods that use observable teacher characteristics (such as experience and credentialing) to improve the precision of VAM teacher measures that can be used across a broad array of VAM approaches. As part of this work, the project will (1)develop methods that use observable teacher characteristics to pool information across teachers as a means of increasing the precision of estimated teacher effects, and (2) use empirical analyses and simulation to assess the gains in precision provided by these methods.

The project will synthesize the findings from its three components in order to propose strategies for providing estimated individual teacher effects that best balance bias reduction and precision.