|Title:||Intended and Unintended Consequences of State High-Stakes Testing: Evidence from Standards-Based Reform in Massachusetts|
|Principal Investigator:||Murnane, Richard||Awardee:||President and Fellows of Harvard College, Graduate School of Education|
|Program:||Evaluation of State and Local Education Programs and Policies [Program Details]|
|Award Period:||3 years||Award Amount:||$450,000|
|Goal:||Efficacy and Replication||Award Number:||R305E100013|
Co-Principal Investigator: John Willet
Purpose: This project will evaluate the impact of high stakes (for students and schools or for schools only) testing in Massachusetts. Massachusetts began administering the Massachusetts Comprehensive Assessment System (MCAS) mathematics and English/language arts examinations in 1998. For the class of 2003, the 10th grade tests became high-stakes exit examinations. Students must pass both tests in order to receive a high school diploma.
Project Activities: Using two quasi-experimental methods (interrupted time series and regression discontinuity) the project will examine: (1) whether the introduction of the exit examinations caused Massachusetts students to substitute the GED for a high school diploma; (2) whether failing the exit exam leads students to pursue a GED versus students who just pass; (3) whether the labeling of students as Basic, Needs Improvement, Proficient, or Advanced based on their 8th grade test scores affects outcomes, such as future achievement, grade retention, absenteeism, and educational attainment; and (4) whether performance on the 8th grade and high school state tests affects students' educational aspirations and whether the effect of failing the 10th grade mathematics examination on the probability of high school graduation (or college attendance) depends on students' prior educational aspirations.
The project will combine data from several sources to carry out the analyses including: (1) the Massachusetts Department of Education's longitudinal database that tracks students throughout their school careers and includes student-level MCAS test results, demographic characteristics, school identifiers, disposition at the time their cohort graduated (e.g., graduated, dropped out, still enrolled) and a range of other information; (2) college attendance data from the National Student Clearinghouse database; (3) the Massachusetts Department of Education's survey given to 8th and 10th graders when they take the MCAS; and (4) state GED testing records.
Products: This work will produce published reports of the secondary data analyses, which will contribute to the knowledge base on the effects of high-stakes tests on a range of student academic outcomes.
Setting: The project will examine records of students in the Massachusetts public school system from 2002 through 2009 and GED testing records for students from 1980 to 2009.
Population: The entire population of students in public schools in Massachusetts during this period will be included in the study with specific analyses focusing on specific years and grades.
Intervention: In 1993, Massachusetts passed legislation that brought standards-based educational reform to the state through standardized testing in grades 3 through 8 with consequences for schools and districts and exit examinations in mathematics and English language arts that high school students must pass in order to graduate. The exit examination requirements first applied to the 2003 graduating cohort.
Research Design and Methods: The project will use two quasi-experimental methods. First, the state imposed the exit exam requirement in 2003 and an interrupted time-series (ITS) approach will be used to determine whether this policy change caused students to substitute the GED for a high school diploma. Second, on the examination itself, students are assigned different performance labels (e.g., Warning/Failure, Needs Improvement, Proficient, or Advanced) based on their underlying test scores. These exogenous cut scores will be used in regression discontinuity (RD) designs to answer a range of questions about the effects of failing the test and of receiving each of the performance labels.
Control Condition: The comparison condition is composed of students whose records fall just on the other side of the exogenous cut point, e.g., the cohorts just before the 2003 graduating cohort for the ITS design and the students who barely pass versus barely fail the examination for the RD design.
Key Measures: The key outcome measures are student educational attainment, educational aspirations, future test performance, retention in grade, and absenteeism. Key predictors include test performance, educational aspirations, and demographics.
Data Analytic Strategy: Non-parametric RD approaches (following Imbens & Lemieux 2008) are used and are to be extended to the case where multiple forcing variables assign students to different treatment conditions. Standard techniques for ITS designs, including a no-treatment comparison group, are used and will be analyzed through an Ordinary Least Squares regression model estimated separately for two independent variables: (1) a continuous variable indicating the number of students in each cohort who took at least one GED examination by a particular age (for example, age 19); and (2) a continuous variable indicating the number of students in each cohort who passed the GED examinations and obtained the credential by a particular age. Descriptive analysis will be used to supplement the causal analysis and provide increased understanding of the factors that predict success in high school and in postsecondary education.
Related IES Projects: The Consequences for High School Students of Failing State Exit Exams: Evidence from Massachusetts (R305A080127)
Journal article, monograph, or newsletter
Papay, J.P., Murnane, R.J., and Willett, J.B. (2014). High–School Exit Examinations and the Schooling Decisions of Teenagers: Evidence From Regression–Discontinuity Approaches. Journal of Research on Educational Effectiveness, 7(1): 1–27.
Papay, J.P., Murnane, R.J., and Willett, J.B. (2016). The Impact of Test Score Labels on Human–Capital Investment Decisions. Journal of Human Resources, 51(2), 357–388.
Papay, J.P., Willett, J.B., and Murnane, R.J. (2011). Extending the Regression–Discontinuity Approach to Multiple Assignment Variables. Journal of Econometrics, 161(2): 203–207.
Papay, J.P., Willett, J.B., and Murnane, R.J. (2015). Income–Based Inequality in Educational Outcomes: Learning From State Longitudinal Data Systems. Educational Evaluation and Policy Analysis, 37(1): 29S–52S.