Co-Principal Investigators: Wulfson, Jeff; Murnane, Richard

Partner Institutions: Brown University, Massachusetts Department of Elementary and Secondary Education

Purpose: The partnership team will evaluate Massachusetts' long-standing policy of requiring students to pass high school exit examinations (HSEE) in core subjects to earn a high school diploma. The team will generate robust causal evidence of the impact of Massachusetts exit examinations administered between 2002 and 2019 on long-term student outcomes including educational attainment, labor market earnings, and criminal justice involvement. During the state's transition to a new assessment (2020-2023), the team will track implementation and short-term student outcomes, to gauge the supports schools provide to help students pass the tests, and to identify any adverse impacts or shortcomings.

Project Activities: Researchers will conduct a historical analysis of exit exams, up to and including the graduating class of 2019, in years 1-3 of the project (2019-2021). Beginning in fall 2019, the team will track implementation and impacts for the new HSEE policy; this analysis will be ongoing for the duration of the project.

Products: Researchers will generate historical evidence on the impacts of HSEEs for key student groups and real-time evidence on the short-term effects and implementation procedures for new HSEEs. The team will share their findings through research briefs targeted toward policymakers, and through working papers and peer-reviewed publications.

Structured Abstract

Setting:The research will take place across the state of Massachusetts.

Sample: Researchers will analyze data on all high school test takers in the state, approximately 70,000 per year, from the class of 2004 through the class of 2023. The total sample of approximately 1,400,000 students will include significant numbers of students who have had difficulty with HSEEs including English learners, students with disabilities, and low-income students.

Intervention: The intervention includes three different generations of the Massachusetts HSEE: the 2002 version, which was the first test students were required to pass; the 2010 version, which increased the math and ELA pass scores and added a science test requirement; and the 2020 version, aligned to the state's College and Career Readiness framework.

Research Design and Methods: Researchers will use a regression discontinuity design (RDD) to evaluate the impacts of test failure. One of the primary interests is the difference in subsequent outcomes for students who fail a test by a narrow margin. The RDD design will show the comparison of outcomes between students with scores just above and below the threshold passing score. Researchers will use a difference-in-differences (DiD) design to evaluate the effect of historical changes to the HSEE policy, such as increases to the threshold passing scores in 2010, as well as an entirely new test beginning in spring 2020. In these models, researchers will assess the effects of a policy change for students with test scores in a range affected by a given policy change. In both sets of models, the researchers will assess whether impacts differ across key demographic groups. In the implementation analysis, researchers will obtain a representative sample of schools across the state and will employ quasi-experimental and descriptive research designs to provide rapid feedback to the state as it fine-tunes implementation of the new HSEE policy.

Control Condition:In the RDD models, the control condition will be students who narrowly passed an HSEE. In the DiD models, the control condition will be students unaffected by a policy change.

Key Measures:The impact evaluation will focus on key student education outcomes, including educational attainments (high school graduation, college-going, college completion), intermediate outcomes (attendance, college remediation), and longer-term life outcomes (criminal justice involvement, labor market participation, earnings). The implementation study will gather data on district and school procedures for supporting students who fail an HSEE in tenth grade.

Data Analytic Strategy: Researchers will useregression models with fixed effects for cohorts and, in some analyses, for high schools.

Cost Analysis: Researchers will conduct a cost analysis (using the ingredients method) to take account of costs associated with implementing HSEEs and their related supports. They will then compare these costs to the benefits of the policy.

Related IES Projects: The Consequences for High School Students of Failing State Exit Exams: Evidence from Massachusetts (R305A080127); Intended and Unintended Consequences of State High-Stakes Testing: Evidence from Standards-Based Reform in Massachusetts (R305E100013).