Regression Discontinuity Designs with Assignment Based on Multiple Rating Scores: Statistical Properties and Issues in the Context of Education Evaluation

Co-Principal Investigator: Reardon, Sean

Four different approaches have been used in the applied literature to analyze program impacts using a multi-rating RD design (MRRD). The multivariate approach models the impact of the intervention as a discontinuity in the multidimensional response (outcome) surface. The centering approach collapses the rating variables into a single variable, thus making it possible to estimate the impact of the intervention using a single-rating RD design. The subset approach also reduces the MRRD design to the simpler single-rating RD design. For example, when two ratings are used, the subset design discards data for individuals who pass one of the pretests and then models the impact of the program as a discontinuity at the threshold for the other pre-test (the reverse can be done as well leading to two separate analyses). The fourth approach pretends that assignment to the program is based on only one of the rating scores and to estimate program impacts using a fuzzy regression discontinuity analysis.

Prior work has shown that all four approaches can produce unbiased impact estimates in theory. However, there are several fundamental differences between them that may affect their relative performance in practice.

Part II of the project examined the relative performance of the approaches in a real world setting, by creating a pseudo-MRDD design from a randomized experiment (the Enhanced Reading Opportunities study), and then comparing the impact estimates produced using the MRRD approaches to the (unbiased) impact estimate yielded by the experimental design.

Two or more baseline covariates in the experiment were selected for use as rating variables; the MRRD dataset was then created by retaining treatment group members who satisfy a cut-off on these rating variables and control group members who do not. The impact estimates obtained from the MRRD approaches (global and local versions) were compared to the benchmark experimental estimate from the random assignment study from which the RD dataset was created. The trade-off between bias and precision was assessed for the different MRRD approaches.

Part III of the project synthesized the lessons learned from Parts I and II into a "best practice" guide for researchers on the MRRD design; recommendations will be illustrated by estimating the effect of Adequate Yearly Progress (AYP) status on student achievement using a school-level dataset with information on AYP status, proficiency rates, school characteristics, and student achievement. The lessons learned from Parts I and II were applied to estimating the effect of AYP status on achievement.