Co-Principal Investigator: Lipscomb, Stephen

Purpose: The Pennsylvania Department of Education (PDE) and Mathematica examined educational inequities in Pennsylvania that have occurred since the disruptions of COVID-19 and evaluated strategies to reduce those inequities.

Project Activities: The project team carried out a survey of Pennsylvania local education agencies (LEAs) and analyzed data from the survey and data from the Pennsylvania Department of Education's student longitudinal data system (along with data from other state agencies) to address four research questions.

• To what extent did COVID-19 and related disruptions of school operations affect student performance and exacerbate inequities on Pennsylvania's statewide assessment in 2021?
• How did different school strategies for dealing with COVID-19 affect educational outcomes, health outcomes, and inequities in both?
• What have been the turnover rates among Pennsylvania educators and administrators during the pandemic, and how has the turnover differed based on characteristics of staff and their schools or districts?
• How did dropout rates change, and what factors were associated with greater likelihood of dropping out during the pandemic?

Key Outcomes: Research on school instruction during COVID-19, school year (SY) 2020–21 (Lipscomb, S., Crigler, F., and Chaplin, D. 2021) found the following:

• In-person learning expanded as the 2020–2021 school year progressed.
• Most students were in LEAs that prioritized students with disabilities for in-person instruction. These LEAs reported that the pandemic made it more difficult to provide special education services.
• Access to reliable internet connection was a larger barrier to remote and hybrid learning than access to digital devices was, especially in urban areas.
• Most students were in LEAs that adopted a wide range of health and safety measures to reduce the spread of COVID-19, although few were in LEAs with regular COVID-19 testing.

Research on student outcomes on the Pennsylvania state assessment in SY 2020–21 and SY 2021–22 (Lipscomb, S., Chaplin, D., Vigil, A., & Matthias, H. 2022 and Lipscomb, S., Chaplin, D., Vigil, A., & Matthias, H. 2023) found the following:

• Students in vulnerable groups and in fully remote learning had lower assessment participation rates than other students in 2021.
• The pandemic led to a substantial reduction in academic proficiency statewide in 2021, reducing academic achievement even more than the results for tested students indicated.
• The pandemic had consistently large negative academic effects across student groups, including for vulnerable groups that had low rates of proficiency before the pandemic.
• LEAs that were fully remote after winter break saw lower proficiency rates in 2021, compared with LEAs that offered fully in-person learning to similar students.
• By spring 2022, academic proficiency rates in grades 5 through 8 in Pennsylvania public schools had regained 6 points of their 10-point drop in English language arts and nearly 5 points of their 13-point drop in math, relative to rates predicted for 2021.

Research on staff attrition outcomes during SY 2020–21 (Lipscomb, S., Lai, I., Chaplin, D., Vigil, A., & Matthias, H. 2022) found the following:

• Five percent of staff working in Pennsylvania LEAs in fall 2020 left the statewide public-school sector by fall 2021, an increase from 4 percent attrition in preceding years.
• Pre-pandemic attrition rates for staff of color were generally higher than for non-Hispanic White staff, and these differences grew during the pandemic except for school administrators, where they narrowed.
• Gaps in attrition rates were also exacerbated during the pandemic for teachers and health and counseling staff early in their careers and in charter schools and for administrators in large LEAs.
• Remote learning was not associated with increased attrition among elementary school teachers.

Research on remote learning outcomes during SY 2020–21 (Lipscomb, S., Chaplin, D., Lai, I., Vigil, A., and Matthias, H. 2023) found the following:

• Vulnerable groups of students were more likely than students overall to attend LEAs where the predominant instructional modes included remote learning.
• Vulnerable groups of students were more likely to be in LEAs where larger shares of the student population had trouble accessing reliable internet.
• More remote learning was associated with lower assessment performance in grades 5 through 8, especially in schools with higher levels of economic disadvantage. It was also associated with lower suspension rates in grades 6 through 12 and did not appear to harm or help graduation rates.
• Remote learning in high school was associated with lower subsequent COVID-19 rates within an LEA's boundary, suggesting it had an important public health benefit.

Structured Abstract

Setting: The project address public school students in Pennsylvania.

Sample: The sample included all Pennsylvania local education agencies (LEAs) from school year (SY) 2015 to SY 2021.

Issue Examined: The project team was interested in examining how COVID-19 and LEA responses to it may have increased educational inequalities among students.

Research Design and Methods: The project team carried out descriptive and regression analyses of data from the Pennsylvania Department of Education student longitudinal data system, Pennsylvania Department of Health records on COVID-19 infection rates, Pennsylvania Department of Human Services (DHS) records on system involvement of students, and data from a project-led survey of Pennsylvania local education agencies (LEAs) on their instructional approaches used during SY 2020–21.

Control Condition: There is no true control condition, but the research team examined variation in LEAs' responses to the disruption caused by COVID-19.

Key Measures: Student academic outcomes (achievement and high school graduation) and characteristics were drawn from the Pennsylvania Information Management System (PIMS). Teacher outcomes (retention from 2019–2020 to 2020–2021 and inactivity during the 2020–2021 school year) and characteristics were also drawn from the PIMS. Information on LEA strategies used to address COVID-19 disruptions were obtained from a survey of 200 LEAs (78 percent response rate) regarding instruction during SY 2020–2021.

Data Analytic Strategy: The research team used descriptive analyses and regression analyses.

State Decision Making: Key PDE leadership and the State Board of Education were briefed on the key project findings. The survey results were shared with 700 Pennsylvania chief school administrators as well as statewide professional associations and other key stakeholders.

Related IES Projects: Pennsylvania Information Management System (PIMS) (R372A060083), Strengthening PIMS Infrastructure to Expand Data Use Capacity (R372A200017), Mid-Atlantic Regional Education Laboratory

Products and Publications

ERIC Citations: Find available citations in ERIC for this award here.

Select Publications:

Lipscomb, S., Chaplin, D., Vigil, A., & Matthias, H. (2022). How the COVID-19 pandemic affected academic proficiency rates in Pennsylvania in 2021: Findings from a predictive model. Cambridge, MA: Mathematica.

Lipscomb, S., Chaplin, D., Lai, I., Vigil, A., and Matthias, H. (2023). Did remote learning lead to different education and health outcomes in Pennsylvania. Cambridge, MA: Mathematica.

Lipscomb, S., Chaplin, D., Vigil, A. and Matthias, H. (2023, March 13). Pennsylvania student proficiency rates rebound partially from COVID-19-related declines.Inside IES Research: Notes from NCER & NCSER.

Lipscomb, S., Chaplin, D., Vigil, A., and Matthias, H. (2022). Pennsylvania's 2022 student proficiency rates show signs of partial recovery. Cambridge, MA: Mathematica.

Lipscomb, S., Crigler, F., and Chaplin, D. (2021) School instruction in Pennsylvania during the COVID-19 pandemic. Cambridge, MA: Mathematica.

Lipscomb, S., Lai, I., Chaplin, D., Vigil, A., & Matthias, H. (2022). Staff attrition from Pennsylvania public schools during the COVID-19 pandemic. Cambridge, MA: Mathematica.