Project Activities
Test scores are ubiquitous in education research aimed at estimating the causal effects of programs, policies, and practices. However, test scores are also complex statistical objects that can create complications for causal analysis. This research project will address the difficulties in "manipulation testing" in regression discontinuity design (RDD) studies that have a test score running variable. Under the What Works Clearinghouse Handbook 5.0, RDD studies must demonstrate the integrity of the forcing variable "(a) institutionally, (b) statistically, and (c) graphically" to "Meet WWC Standards Without Reservations". However, the way test scores are commonly constructed creates unsmooth distributions that will fail both statistical and graphical check, even absent any "manipulation" that interferes with validity of an RDD study. The study will address this current conundrum in RDD research by exploring systematically how common the problem is and the viability of different alternate solutions for establishing the integrity of test score running variables.
Research plan
The research plan for this study has two overarching objectives: 1) to formally document the extent to which manipulation testing in RDDs that use test score running pose a problem for education research and 2) to develop and evaluate potential solutions to this problem. The objectives will be achieved through a combination of literature review, simulation study, and analysis of real-world test score data. In the literature review, this project will use the Education Resources Information Center (ERIC), 2) the National Bureau of Economic Research Economics of Education program working papers, 3) the Annenberg EdWorkingPapers, and 4) ProQuest database dissertations to identify all test score RDDs going back five years, and then 45 will be selected at random to read in depth. The results of this review will be used to provide a broad overview of the state of the field, which will be used to inform a simulation study of test score distributions and analysis of real data from the Massachusetts MCAS and the New York Regent Exam.
Career plan
The career development plan will support the PI's research activities and a set of focused career development activities aimed at three domains: 1) expanding quantitative methods (for causal inference and psychometrics) content knowledge, 2) improving the PIs ability to disseminate methods products and code, 3) acquiring the skills needed to manage junior research staff on projects. Two experienced mentors, Drs. Weiss and Soland, will guide the PI through regular meetings to obtain feedback and general advice, formal trainings and workshops at conferences, and self-directed learning activities.
People and institutions involved
IES program contact(s)
Project contributors
Products and publications
Products: Products for this project will describe why scaled score distributions pose a problem for manipulation testing and provide advice for researchers on how to navigate this problem. The project will result in publications and presentations as well as other dissemination products (e.g., a workshop and R code) that will reach methodologists, students, and the broader research community.
ERIC Citations: Find available citations in ERIC for this award here.
Questions about this project?
To answer additional questions about this project or provide feedback, please contact the program officer.
