Co-Principal Investigators: Corn, Jeni; Pond, Karl; Fuller, Sarah C.; Lauen, Douglas; Bastian, Kevin

Related Network Teams: In FY22, this project joined the Prekindergarten Through Grade 12 Recovery Research Network as a research team. This network includes the following other projects — PreK–12 RESTART Network Lead (Research on Education Strategies to Advance Recovery and Turnaround) (R305X220009), Evaluation of the Learning Renewal-Social Emotional Learning Programs (LR-SEL) for Supporting Pandemic Recovery with Illinois State Board of Education (ISBE) (R305X220015), A Researcher-Practitioner Partnership to Improve Pandemic Recovery Efforts in California (R305X220028), Mitigating the Impact of the COVID-19 Pandemic Through Curriculum-Based Approaches to Learning Acceleration in Grades K–2 in Chicago (R305X220033), Lessons from the Pandemic: The Effects of Remote Instructional Delivery and Recovery Strategies on Student Outcomes (R305A230400)

Purpose: The purpose of this project is to examine the effects of the pandemic on educational outcomes in North Carolina and the impact of school extension programs on school re-engagement and academic recovery. Early evidence suggests that the pandemic had substantial negative impacts on student outcomes, with disproportionate impacts on historically marginalized groups. In North Carolina, the Office of Learning Recovery and Acceleration (OLR) is using federal funding to direct a set of school extension activities intended to promote academic recovery and close achievement gaps. This project will provide evidence to support decision making around the design and continuation of school extension activities and success in the use of these activities in closing achievement gaps created or widened by the pandemic.

Project Activities: This work aims to increase student re-engagement in school and academic recovery by 1) examining the short, medium, and long-term effects of the pandemic on statewide trends in student outcomes—overall and by subgroup—during the 2020–21 to 2023–24 school years, 2) descriptively exploring variation in the implementation and design of school extension programs sponsored by the OLR across a range of districts and charter schools, and 3) analyzing the impact of school extension programs on student re-engagement and academic recovery across student subgroups by program type, dosage, and design features.

Products: The project team will produce evidence to aid decision-making by identifying areas of continuing need and providing data on the effects of school extension programs on student outcomes. The research team will summarize interim and final findings in 3 annual reports to state policymakers, 3–4 policy briefs, conference presentations, and 3–4 peer reviewed publications. The research team will present findings to the North Carolina State Board of Education and the North Carolina Department of Public Instruction as well as disseminating promising practices to local school boards and superintendents across the state.

Structured Abstract

Setting: This project takes place in the state of North Carolina.

Population/Sample: This project will use data from all public school students in all 115 of the North Carolina school districts and over 200 charter schools across the state from the 2015–16 through 2023–24 school years.

Key Issue, Program or Policy: The COVID-19 pandemic has had detrimental effects on school enrollment, student learning, credit recovery, and educational attainment in North Carolina. Early evidence suggests that these impacts are exacerbating existing inequalities in educational outcomes between student subgroups of interest. In response to the pandemic, the NC Department of Public Instruction (NCDPI) created the Office of Learning Recovery and Acceleration (OLR) to address the learning loss associated with the disruptions to the educational system. A central component of the work of the OLR is a set of school extension programs that provide additional grade level instruction, socioemotional learning support, and enrichment through summer programs, after school programs, and individual tutoring.

Research Design and Methods: The research design for this study is an iterative mixed methods design which allows for adaptation of the qualitative and quantitative approaches in subsequent rounds of data collection and analysis to adapt to new knowledge gained from previous rounds of analysis and changes in the program implementation. The analyses for this grant will include descriptive interrupted time series analyses to examine the change in student outcomes during the years of the pandemic, a limited implementation analysis aimed at providing data on program participation and the variations in design of OLR sponsored school extension programs, and an analysis of the impact of OLR sponsored programs overall and across different subgroups.

Control Condition: There will be no true control condition, but the research team will examine variations across student groups and some analyses will create comparison conditions of statistically equivalent students who did not participate in school extension programs.

Key Measures: Outcome measures drawn from the State Longitudinal Data System (SLDS) will include student engagement measures — attendance, course grades, and school discipline; student achievement measures — iReady assessments, mCLASS early literacy scores, state end-of-grade and end-of-course test scores, and ACT scores; and school progress and completion measures — credit recovery, grade retention, on-track in high school, drop out, graduation, and college enrollment. The study will also collect measures of program implementation through local program planning documents, surveys, and interviews. Analyses will also include SLDS measures of student and school characteristics.

Data Analytic Strategy: The quantitative analysis of pandemic impacts will include descriptive interrupted time series analysis to look at changes in student outcome measures in each pandemic and post-pandemic school year (2019–20 to 2023–24) compared to pre-pandemic student outcomes (2015–16 to 2018–19). The analysis of school extension program implementation will begin with a descriptive analysis of participation in school extension programs by student and school characteristics. Qualitative analysis of implementation will involve document review, interviews, and coding documents and interviews for common and discrepant themes. To focus on program impacts, the research team will use a quasi-experimental design (propensity score weighting) to perform an impact analysis of the effects of the OLR sponsored school extension programs overall, across subgroups of students, and by program design features.

State Decision Making:

The results of this project will provide actionable information to the OLR and the larger education community to support decision-making about designing, targeting, and continuing school extension programs to support student re-engagement and academic recovery.

The first research question on the impacts of the pandemic across a range of student engagement, achievement, and school progression outcomes will allow policymakers to set recovery targets that groups of students must achieve to be considered to have made a resilient or transformative recovery. This work will also identify the groups for whom the pandemic had the largest impacts as well as the groups who experienced the largest and most persistent gaps in outcomes.

The second research question will provide crucial Public School Unit (PSU) level implementation evidence to decisionmakers at NCDPI so they can adjust program targeting and offerings to ensure that the programs fully serve the needs of at-risk students.

The third research question will assist the state and individual PSUs in making decisions about the design of their school extension programs and the students who are best served by the programs in different capacities. These results will also support decision-making around continuation of the programs in the longer term.

Related IES Projects:

This project builds on two SLDS grants awarded to NCDPI by IES in 2007 (North Carolina Common Education Data Analysis and Reporting System (NC CEDARS)) and 2012 (The NC P–20W SLDS Project: Creating a Preschool to Workforce Statewide Longitudinal Data System in North Carolina).