Mentors: Soland, James; Weiss, Michael

Purpose: Test scores are ubiquitous in education research aimed at estimating the causal effects of programs, policies, and practices. However, test scores are also complex statistical objects that can create complications for causal analysis. The purpose of this project is to develop the Principal Investigator's (PI's) career as an education methodologist who combines the insights and frameworks of psychometrics and causal inference to tackle the methodological complications of doing causal analysis with test scores. This career development includes a research project that applies psychometric knowledge to a practical problem faced by causal evaluation researchers. Specifically, this research project will address the difficulties in "manipulation testing" in regression discontinuity design (RDD) studies that have a test score running variable. Under the What Works Clearinghouse Handbook 5.0, RDD studies must demonstrate the integrity of the forcing variable "(a) institutionally, (b) statistically, and (c) graphically" to "Meet WWC Standards Without Reservations". However, the way test scores are commonly constructed creates unsmooth distributions that will fail both statistical and graphical check, even absent any "manipulation" that interferes with validity of an RDD study. The study will address this current conundrum in RDD research by exploring systematically how common the problem is and the viability of different alternate solutions for establishing the integrity of test score running variables.

Research Plan: The research plan for this study has two overarching objectives: 1) to formally document the extent to which manipulation testing in RDDs that use test score running pose a problem for education research and 2) to develop and evaluate potential solutions to this problem. The objectives will be achieved through a combination of literature review, simulation study, and analysis of real-world test score data. In the literature review, this project will use the Education Resources Information Center (ERIC), 2) the National Bureau of Economic Research Economics of Education program working papers, 3) the Annenberg EdWorkingPapers, and 4) ProQuest database dissertations to identify all test score RDDs going back five years, and then 45 will be selected at random to read in depth. The results of this review will be used to provide a broad overview of the state of the field, which will be used to inform a simulation study of test score distributions and analysis of real data from the Massachusetts MCAS and the New York Regent Exam.

Career Development Plan: The career development plan will support the PI's research activities and a set of focused career development activities aimed at three domains: 1) expanding quantitative methods (for causal inference and psychometrics) content knowledge, 2) improving the PIs ability to disseminate methods products and code, 3) acquiring the skills needed to manage junior research staff on projects. Two experienced mentors, Drs. Weiss and Soland, will guide the PI through regular meetings to obtain feedback and general advice, formal trainings and workshops at conferences, and self-directed learning activities.

Products and Publications

Products: Products for this project will describe why scaled score distributions pose a problem for manipulation testing and provide advice for researchers on how to navigate this problem. The project will result in publications and presentations as well as other dissemination products (e.g., a workshop and R code) that will reach methodologists, students, and the broader research community.

ERIC Citations: Find available citations in ERIC for this award here.