Skip Navigation
Funding Opportunities | Search Funded Research Grants and Contracts

IES Grant

Title: Sensitivity Analysis—If We're Wrong, How Far Are We from Being Right?
Center: NCER Year: 2011
Principal Investigator: Hill, Jennifer Awardee: New York University
Program: Statistical and Research Methodology in Education      [Program Details]
Award Period: 3 years Award Amount: $928,537
Type: Methodological Innovation Award Number: R305D110037

Co-Principal Investigator: Marc Scott

This project will extend existing and develop new methods for sensitivity analyses that can be used to quantify the uncertainty about causal inferences made when strong and often assumptions are required that are not testable. An example of this is when observational data is used or when randomized experiments suffer from missing data or non-compliance with assignment. Although reasonable strategies for sensitivity analyses already exist to address some of these assumptions, most remain severely underdeveloped and, in some cases, difficult for education researchers to use. This project intends to provide education researchers with accessible and user-friendly methods to test the key assumptions underlying their causal inferences. With these methods, researchers will be able to gauge how far their estimates might be from the truth.

Sensitivity analysis is an umbrella term that encapsulates any of a variety of different methods that assess the degree to which inferences might be altered by changes in structural or parametric assumptions. These strategies allow for an assessment of uncertainty surrounding the assumptions used and the resulting inferences. Even when analyses rely on assumptions that are either not testable or difficult to assess, there is still information available. This information can be found both in the data and in the utilization of substantive knowledge of the area. This information can be used to help gauge how reasonable these assumptions are and to estimate how inferences might change if these assumptions were violated in specific ways.

Specifically, the project intends to extend and develop methods to explore the sensitivity of inferences to deviations from the required assumptions in the context of observational studies and randomized experiments. For observational studies, the project will focus on methods to explore the sensitivity of inferences to: (1) one or more omitted confounders; (2) mis-specification of the model for the response surface; and (3) lack of overlap of covariate distributions across treatment and comparison groups. For randomized experiments or natural experiments, the project will focus on methods to assess the sensitivity of inferences to: (1) deviations from pure randomization, and (2) departures from the exclusion restriction when attempting to correct for non-compliance using instrumental variables methods.

In addition, the project will develop practical guidelines for using sensitivity analyses in applied settings by testing the efficacy of competing methods in empirical settings with known answers and identifying better benchmarks for the plausibility of sensitivity analysis parameters. User-friendly software to implement these strategies will be developed to make them accessible to education researchers. Also, the project will develop procedures for representing results graphically to facilitate interpretation and will build these procedures into the software.


Journal article, monograph, or newsletter

Carnegie, N.B., Harada, M., and Hill, J.L. (2016). Assessing Sensitivity to Unmeasured Confounding Using a Simulated Potential Confounder. Journal of Research on Educational Effectiveness, 9(3), 395–420.

Dorie, V., Harada, M., Carnegie, N.B., aand Hill, J. (2016). A Flexible, Interpretable Framework for Assessing Sensitivity to Unmeasured Confounding. Statistics in Medicine, 35(20), 3453–3470.

Dorie, V., Hill, J., Shalit, U., Scott, M., and Cervone, D. (2017). Automated Versus Do-It-Yourself Methods for Causal Inference: Lessons Learned From a Data Analysis Competition. arXiv preprint arXiv:1707.02641.

Hill, J., and Su, Y.-S. (2013). Assessing Lack of Common Support in Causal Inference Using Bayesian Nonparametrics: Implications for Evaluating the Effect of Breastfeeding on Children's Cognitive Outcomes. Annals of Applied Statistics, 7(3): 1386–1420.

Kern, H.L., Stuart, E.A., Hill, J., and Green, D.P. (2016). Assessing Methods for Generalizing Experimental Impact Estimates to Target Populations. Journal of Research on Educational Effectiveness, 9(1), 103–127.