Estimation and Inference in Education Research when Actions by Participants Impact Validity and Availability of Data

When an educational program offered by a school district is oversubscribed, the district is likely to determine access to the program based on lotteries that condition on priority classes. These lotteries provide random variation in participation and, therefore, can be used to estimate treatment effects. However, lotteries only provide randomization at the first stage of the program (i.e., the participation stage). As a consequence, they can be used to estimate the effect of the program participation on retention. If the program is successful in retaining students, it potentially creates a differential attrition problem in the research design. Families who do not win lotteries are more likely to enroll their children outside the public school system. This situation can be particularly problematic for large urban public schools that compete against a variety of charter, private, and suburban schools. As a consequence, researchers face differential attrition rates among the treatment and control group. The project compared the results of experimental estimators to a set of non-experimental estimators (such as fixed effect estimators, matching estimators, and instrumental variable estimators) when applied to students with low or high attrition.