Purpose: The project developed methods for estimation and inference in education research when actions by participants impact validity and availability of data. Specifically, the researchers addressed the impacts of participant actions in two research designs: differential attrition in lottery designs and manipulation of the assignment variable in regression discontinuity designs. The researchers used data from the Pittsburgh Public School’s magnet program to address attrition in lottery designs and data from Pittsburgh’s gifted education program to address manipulation in regression discontinuity designs.

Project Activities: When an educational program offered by a school district is oversubscribed, the district is likely to determine access to the program based on lotteries that condition on priority classes. These lotteries provide random variation in participation and, therefore, can be used to estimate treatment effects. However, lotteries only provide randomization at the first stage of the program (i.e., the participation stage). As a consequence, they can be used to estimate the effect of the program participation on retention. If the program is successful in retaining students, it potentially creates a differential attrition problem in the research design. Families who do not win lotteries are more likely to enroll their children outside the public school system. This situation can be particularly problematic for large urban public schools that compete against a variety of charter, private, and suburban schools. As a consequence, researchers face differential attrition rates among the treatment and control group. The project compared the results of experimental estimators to a set of non-experimental estimators (such as fixed effect estimators, matching estimators, and instrumental variable estimators) when applied to students with low or high attrition.

Regression discontinuity designs (RDD) can often be used where admission to an educational program is determined by clearly stated, transparent rules instead of the discretion of administrators. RDD offers a quasi-experimental method that can provide estimators of causal relationships in the absence of randomized trials. Reliable estimates can be obtained with limited attention to statistical issues related to selection of observables or un-observables. Researchers have identified a number of potential pitfalls associated with RDD. One of the most severe problems encountered in RDD design is potential for manipulation of the assignment variable which will lead to inconsistent estimators. This project developed new estimators that can be used to recover the relevant treatment effects in the presence of some types of manipulation of the criterion variable.

Related IES Projects: Determinants of Student Outcomes in an Urban School District: Educational Interventions and Family Choices (R305A070117)

Products and Publications

Journal article, monograph, or newsletter

Davis, B., Engberg, J., Epple, D., Sieg, H., and Zimmer, R. (2013). Bounding the Impact of a Gifted Program on Student Retention Using a Modified Regression Discontinuity Design. Annals of Economics and Statistics/ANNALES D'ÉCONOMIE ET DE STATISTIQUE, 10–34.

Engberg, J., Epple, D., Imbrogno, J., Sieg, H., and Zimmer, R. (2014). Bounding the Treatment Effects of Education Programs That Have Lotteried Admission and Selective Attrition. Journal of Labor Economics, 32(1): 27–63.

Zimmer, R., and Engberg, J. (2016). Can Broad Inferences be Drawn From Lottery Analyses of School Choice Programs? An Exploration of Appropriate Sensitivity Analyses. Journal of School Choice, 10(1), 48–72.

Working paper

Davis, B., Engberg, J., Epple, D.N., Sieg, H., and Zimmer, R. (2010). Evaluating the Gifted Program of an Urban School District Using a Modified Regression Discontinuity Design (NBER 16414). Cambridge, MA: National Bureau of Economic Research Working Paper.

Engberg, J., Epple, D., Imbrogno, J., Sieg, H., and Zimmer, R. (2009). Estimation of Causal Effects in Experiments with Multiple Sources of Noncompliance (No. w14842). Cambridge, MA: National Bureau of Economic Research.