Accountability Community of Practice

REL Mid-Atlantic is partnering with the New Jersey Department of Education (NJDOE) to evaluate the state’s current accountability models and methods for tracking progress and achievement in English language proficiency (ELP). At the student level, the project team is analyzing how well NJDOE’s current ELP student growth model aligns with observed growth trajectories in the state, particularly in the wake of the COVID-19 pandemic. At the school level, the project team is seeking to evaluate NJDOE’s school accountability measure as a valid and reliable indicator of a school’s contribution to students’ ELP growth. At both the student and the school levels, NJDOE seeks to collect evidence that may inform its future decisionmaking about how best to set ambitious achievement standards for students and schools, identify students and schools that need support to meet these standards, and provide meaningful support and guidance to schools and educators whose students are identified for support.

This project aimed to increase the robustness of the Pennsylvania Department of Education’s (PDE’s) accountability system to measurement error, thereby improving its accuracy. The REL team applied a Bayesian statistical model to historical subgroup data and compared the reliability of the stabilized Bayesian proficiency rates to the un-stabilized proficiency rates typically used to inform Targeted Support and Improvement (TSI) and Additional Targeted Support and Improvement (ATSI) determinations. Stabilization substantially improved statistical reliability, especially for small subgroups. Education agencies seeking to make the best use of data on subgroups in schools (or any data with small numbers of students) should consider stabilizing their measures to increase reliability and reduce the risk of being fooled by random variation.

Final report: Stabilizing Subgroup Proficiency Results to Improve the Identification of Low-Performing Schools.

REL Mid-Atlantic worked with the New Jersey Department of Education to ensure that the measures of school-level English language proficiency (ELP) used in its ESSA accountability were reliable and valid in the wake of COVID-related assessment disruptions and for the long term. The REL provided support for the design of diagnostic analyses to assess potential biases in the schoolwide measures of ELP due to testing disruptions in 2020 and 2021 and assistance in interpreting the findings. The REL also provided information about other states’ English learner accountability practices as well as technical specifications to weight existing data on ELP measures using a sliding scale approach in order to be representative of each school’s full population of English learners.

The Accountability Community of Practice worked with NJDOE to explore Bayesian methods as applied to measures used in the state’s school accountability system. The project provided training, code and other coding resources, and an educator-friendly infographic to NJDOE technical program support staff. Short-term outcomes included a better understanding of Bayesian methods and measures, distribution of an infographic on Bayesian stabilization, and improved capacity to implement refinements to the NJDOE accountability system. Medium-term outcomes included a potential to increase the reliability of school-performance measures used in NJ’s accountability system and successful communication of the merits of those measures to NJ educators.

Infographic: Stabilizing School Performance Measures for Accuracy and Equity

REL Mid-Atlantic provided DC Public Charter School Board (PCSB) with technical support as it developed a new framework of school performance measures. The REL advised PCSB about issues related to the reliability, validity, and robustness of measures; and explored implications of different approaches for combining and weighting the measures—with particular attention to PCSB’s desire to promote equity in its framework. The project team provided this feedback to DC PCSB over the course of 5 virtual sessions during the first half of 2023 as they worked to establish a new framework of school performance measures.

Blog post: Using Tailored Supports to Advance Educational Equity

The New Jersey Department of Education wanted to ensure that the measures of school performance used in its ESSA accountability system are reliable, both in the wake of COVID-related assessment disruptions and for the long term. Recognizing that measurement error could challenge reliability, the state explored the potential of a Bayesian stabilization approach to improve the accuracy of accountability measures. The REL team applied Bayesian hierarchical modeling to historical subgroup-by-school-level data to stabilize accountability measures. Read the final report: https://ies.ed.gov/use-work/resource-library/report/descriptive-study/stabilizing-school-performance-indicators-new-jersey-reduce-effect-random-error.