The TCTS project will build the capacity of SDP’s Offices of DEI, Research and Evaluation, and Academic Support to monitor the progress of the CLIF initiative as well as improve its design and implementation. The REL’s technical support will begin in June 2022 and continue through July 2023. The project will include the following activities: a review of existing DEI-related data collection instruments, production of a landscape analysis, and collaboratively determining how existing measures could be adopted or adapted to monitor CLIF implementation; co-developing and supporting SDP’s pilot of test measure(s); and co-designing survey items, conducting cognitive interviews, and leading a technical support session to refine survey items.

This TCTS project will include coaching for central office staff involved in leading and supporting the Equity Partners Fellowship. Coaching will begin in August 2022 and continue throughout the 2022/23 school year to help SDP leaders in the DEI office refine the program and strengthen the supports it will offer to future Equity Fellows, particularly around their action research projects. Coaching will also include modeling the data collection process to support ongoing data informed continuous improvement of the program.

The REL and NJDOE are collaborating to develop and facilitate a coaching series for 10 select New Jersey school districts to build their capacity to make data-informed decisions and implement research-based practices that promote increased hiring of teachers of color throughout the state.

This project aims to increase the robustness of the Pennsylvania Department of Education’s (PDE’s) accountability system to measurement error, thereby improving its accuracy. The REL team will apply a Bayesian model to historical subgroup data and compare the reliability of the stabilized Bayesian proficiency rates to the un-stabilized proficiency rates typically used to inform Targeted Support and Improvement (TSI) and Additional Targeted Support and Improvement (ATSI) determinations. If the stabilized subgroup rates prove more reliable than the corresponding un-stabilized subgroup rates, REL staff will then apply the Bayesian approach to 2022 proficiency data, allowing PDE to include those results into the TSI and ATSI determinations in late 2022.

The Accountability Community of Practice is working with NJDOE to ensure that the measures of school ELP used in its ESSA accountability are reliable and valid in the wake of COVID-related assessment disruptions and for the long term. The REL is providing support for the design of diagnostic analyses to assess potential biases in the schoolwide measures of English language proficiency (ELP) due to missing data from testing disruptions in 2020 and 2021, assistance in interpreting the findings from the proposed diagnostic analyses, providing technical specifications to impute missing data and weight existing data on ELP measures to be representative of each school’s full population of English learners, and coaching and dissemination support on how to effectively describe the proposed alternative ELP indicator weights system and the techniques used to handle missing student-level ELP data for a non-technical audience and stakeholders.

The Accountability Community of Practice is working with NJDOE to explore Bayesian methods, and to evaluate whether using Bayesian methods would improve the reliability of the accountability data, especially when applied to smaller subgroups. The project will provide training, code and other coding resources, and an educator-friendly infographic to NJDOE technical program support staff. Short-term outcomes include a better understanding of Bayesian methods and measures, distribution of an infographic on Bayesian stabilization, and improved capacity to implement refinements to the NJDOE accountability system. Medium-term outcomes include an increase in the reliability of school-performance measures used in NJ’s accountability system, and successful communication of the merits of those measures to NJ educators.

Under this project, REL Mid-Atlantic is providing DC Public Charter School Board (PCSB) with technical support as it develops a new framework of school performance measures. We are advising PCSB about issues related to the reliability, validity, and robustness of measures; and exploring implications of different approaches for combining and weighting the measures—with particular attention to PCSB’s desire to promote equity in its framework. The project team will provide this feedback to DC PCSB over the course of 3 to 4 virtual sessions during the first half of 2023 as PCSB seeks to establish its new framework of school performance measures.

This TCTS project will focus on the new system’s observation component. The project will help the Delaware Department of Education (DDOE) collect and learn from data from the two pilot years of the new system (2021/22 and 2022/23) as DDOE prepares to implement the system statewide in 2023/24. REL staff will partner with DDOE to help them collect and make sense of data on teachers’ and school leaders’ perceptions of the system’s observation component, factors supporting and limiting effective implementation of the observation component, and needed changes or additional support.

REL Mid-Atlantic will conduct a study to support Delaware Department of Education’s (DDOE’s) learning from the pilot implementation of its new teacher observation tool in preparation for statewide implementation of the tool. In the short term, the goal of the project is to better understand teachers’ ratings using the new system by learning (1) how consistently the indicators in the tool measure each of the three performance areas (the tool’s reliability); (2) the extent to which ratings on each of the nine indicators and three performance areas vary across teachers; (3) how strongly teachers’ ratings are associated with their estimated contributions to student achievement growth (the tool’s validity); and (4) whether ratings differ based on the characteristics of teachers or their students (potential bias in the ratings). Next, a second short-term goal is to use the information about reliability, validity, variation, and bias to suggest changes needed to facilitate effective implementation so that DDOE’s evaluation system can support teachers in improving their instructional practices. The study will also support DDOE’s long-term goal of implementing a teacher observation tool that reliably and validly assesses teachers’ instructional performance and effectively distinguishes performance to help teachers improve statewide by 2025-26. The ultimate goal of this project is to improve student outcomes.

The project will consist of three components that aim to make school climate data in Pennsylvania more useful and actionable. For this study, school climate refers to student and school staff perceptions of the quality of the school environment, specifically the quality of educator-student and peer relationships, support for learning and academic engagement, and the degree of safety and respect experienced within the school environment. First, REL staff will assess the psychometric properties of the elementary school climate survey, which was updated based on recommendations from analyses under the last REL contract. Second, to better understand the relationship between the COVID-19 pandemic and school climate, REL staff will assess changes in student and staff perceptions of school climate for middle and high schools before and during the pandemic. REL staff will also examine the association between the percentage of remote and hybrid instruction during the 2020–2021 school year and school climate in the 2021–2022 school year. Third, REL staff will work with PDE to develop a consistent scoring approach for the school climate reports that are generated for individual schools from elementary through high schools. REL staff will also help PDE modify its computer programs that analyze school climate data and provide score reports to schools. One of the goals of this project is to build Pennsylvania Department of Education’s capacity to support local education agencies (LEAs) and schools in using school climate data to understand needs and identify programs to improve the supportiveness of the school learning environment. The project also aims to increase the likelihood that LEAs and schools view the climate survey as useful, so REL staff will track school participation in the climate survey as a medium-term outcome.

The project team is working with Pennsylvania Department of Education (PDE)’s Office for Safe Schools to provide research-based support in building (1) a logic model to connect initiatives that foster supportive learning environments with measurable outcomes, (2) accessible materials, such as an infographic or fact sheet, on evidence-based practices for fostering supportive learning environments for students and staff and 3) a toolkit to support the Intermediate Unit (IU) social emotional wellness leads to assess, plan, and measure social-emotional initiatives and the impact towards creating more supportive learning environments and better social-emotional wellness for students and staff.The goals of this project are to (1) increase PDE staff’s understanding of how to support Intermediate Unit (IU), local education agency (LEA), and school staff in building supportive learning environments for students and staff; (2) deepen PDE staff’s and IU liaisons’ understanding of evidence-based practices and interventions that can support student and staff social-emotional wellness; and (3) build all partners’ understanding of Pennsylvania’s initiatives about supportive learning environments, including how the initiatives relate to one another and the extent to which current evidence suggests they may foster supportive learning environments. The logic models and information on evidence-based programs and practices will be important tools for PDE and other partners when making decisions about how to use resources (including identifying initiatives to add, adjust, or scale).

NJDOE would like to ensure that the measures of school performance used in its ESSA accountability system are reliable, both in the wake of COVID-related assessment disruptions and for the long term. Recognizing that measurement error could challenge reliability, the state is exploring the potential of a Bayesian stabilization approach to improve the accuracy of accountability measures. In an existing project, REL staff are providing training to NJDOE staff on how to implement Bayesian stabilization. In this complementary project, the REL team will design and test Bayesian stabilization models for a public report that will document the methods and their implications. We will apply Bayesian hierarchical modeling to historical subgroup-by-school-level data to stabilize accountability measures, with a short-term goal for NJDOE to develop a better understanding of how modifications to its accountability measures could increase reliability and accuracy.

Under this project, REL Mid-Atlantic is providing the DC Public Charter School Board (PCSB) with technical support as it develops a new framework of school performance measures. We are advising PCSB about issues related to the reliability, validity, and robustness of measures; and exploring implications of different approaches for combining and weighting the measures—with particular attention to PCSB’s desire to promote equity in its framework. The project team will provide this feedback to DC PCSB over the course of 3 to 4 virtual sessions during the first half of 2023 as PCSB seeks to establish its new framework of school performance measures.