REL Mid-Atlantic is partnering with the New Jersey Department of Education (NJDOE) to evaluate the state’s current accountability models and methods for tracking progress and achievement in English language proficiency (ELP). At the student level, the project team is analyzing how well NJDOE’s current ELP student growth model aligns with observed growth trajectories in the state, particularly in the wake of the COVID-19 pandemic. At the school level, the project team is seeking to evaluate NJDOE’s school accountability measure as a valid and reliable indicator of a school’s contribution to students’ ELP growth. At both the student and the school levels, NJDOE seeks to collect evidence that may inform its future decisionmaking about how best to set ambitious achievement standards for students and schools, identify students and schools that need support to meet these standards, and provide meaningful support and guidance to schools and educators whose students are identified for support.

Education agencies need good measures of school performance to know which schools and subgroups of students need support the most. Identifying subgroups most in need has been a long-standing challenge due to the instability of conventional measures for small subgroups of students. With respect to small subgroups, states have faced a difficult tradeoff between accuracy and equity: Excluding the smallest subgroups from accountability makes these students and their needs invisible, compromising equity; but including small subgroups risks making decisions based on unreliable data due to the large amount of random error implicit in small numbers of students. Key findings from our 2023 report, using data from the Pennsylvania Department of Education, demonstrate how Bayesian stabilization increases the reliability of school performance measures, especially for small subgroups of students—making it possible to simultaneously promote accuracy and equity. But not every state has the internal expertise or resources needed to implement stabilization. To make stabilization available to state and local education agencies across the country, we’re developing a free tool, hosted by the U.S. Department of Education, that will allow them to stabilize their own performance measures. The Accuracy for Equity (A4E) tool will be useful not only for improving the accuracy of accountability measures, but also for diagnostic purposes in support of equitable outcomes, by giving states, districts, and schools better information on student groups most in need of support.

This project aimed to increase the robustness of the Pennsylvania Department of Education’s (PDE’s) accountability system to measurement error, thereby improving its accuracy. The REL team applied a Bayesian statistical model to historical subgroup data and compared the reliability of the stabilized Bayesian proficiency rates to the un-stabilized proficiency rates typically used to inform Targeted Support and Improvement (TSI) and Additional Targeted Support and Improvement (ATSI) determinations. Stabilization substantially improved statistical reliability, especially for small subgroups. Education agencies seeking to make the best use of data on subgroups in schools (or any data with small numbers of students) should consider stabilizing their measures to increase reliability and reduce the risk of being fooled by random variation.

Final report: Stabilizing Subgroup Proficiency Results to Improve the Identification of Low-Performing Schools.

REL Mid-Atlantic worked with the New Jersey Department of Education to ensure that the measures of school-level English language proficiency (ELP) used in its ESSA accountability were reliable and valid in the wake of COVID-related assessment disruptions and for the long term. The REL provided support for the design of diagnostic analyses to assess potential biases in the schoolwide measures of ELP due to testing disruptions in 2020 and 2021 and assistance in interpreting the findings. The REL also provided information about other states’ English learner accountability practices as well as technical specifications to weight existing data on ELP measures using a sliding scale approach in order to be representative of each school’s full population of English learners.

The Accountability Community of Practice worked with NJDOE to explore Bayesian methods as applied to measures used in the state’s school accountability system. The project provided training, code and other coding resources, and an educator-friendly infographic to NJDOE technical program support staff. Short-term outcomes included a better understanding of Bayesian methods and measures, distribution of an infographic on Bayesian stabilization, and improved capacity to implement refinements to the NJDOE accountability system. Medium-term outcomes included a potential to increase the reliability of school-performance measures used in NJ’s accountability system and successful communication of the merits of those measures to NJ educators.

Infographic: Stabilizing School Performance Measures for Accuracy and Equity

REL Mid-Atlantic provided DC Public Charter School Board (PCSB) with technical support as it developed a new framework of school performance measures. The REL advised PCSB about issues related to the reliability, validity, and robustness of measures; and explored implications of different approaches for combining and weighting the measures—with particular attention to PCSB’s desire to promote equity in its framework. The project team provided this feedback to DC PCSB over the course of 5 virtual sessions during the first half of 2023 as they worked to establish a new framework of school performance measures.

Blog post: Using Tailored Supports to Advance Educational Equity