Co-Principal Investigators: Capers, K. Jurée; Darling-Aduana, Jennifer

Related Network Teams: In FY23, this project joined the Prekindergarten Through Grade 12 Recovery Research Network as a research team. This network includes the following other projects — PreK–12 RESTART Network Lead (Research on Education Strategies to Advance Recovery and Turnaround) (R305X220009), Assessing the Long-Term Impacts of School Extension Programs on Student Re-Engagement and Learning Recovery; Mitigating the Impact of the COVID-19 Pandemic Through Curriculum-Based Approaches to Learning Acceleration in Grades K–2 in Chicago (R305X220033), A Researcher-Practitioner Partnership to Improve Pandemic Recovery Efforts in California (R305X220028), Evaluation of the Learning Renewal-Social Emotional Learning Programs (LR-SEL) for Supporting Pandemic Recovery with Illinois State Board of Education (ISBE) (R305X220015), Assessing the Long-Term Impacts of School Extension Programs on Student Re-Engagement and Learning Recovery (R305S220003)

Purpose: The combination of family disruptions, closures of schools, and rapid transition to remote learning resulting from the COVID-19 pandemic has substantially reduced achievement growth for many students, particularly students from traditionally marginalized populations. What is not currently known, however, is why some students fared better than others in a remote learning environment. In response, state education agencies and local school districts are implementing a variety of strategies to promote recovery and accelerate student learning, yet there is great uncertainty as to what programs will be most effective and for whom. This project will use rich longitudinal data on students and teachers in three Atlanta metro-area districts that collectively serve nearly a quarter million students to examine both the factors associated with higher achievement growth in remote instruction and the relationship between specific recovery efforts and multiple student outcomes. The findings will provide not only immediate benefits to education agencies working to accelerate student learning but also important insights to guide future research and practice as remote learning becomes a more important element in the post-pandemic educational landscape.

Project Activities: The researchers will survey principals and parents and link survey responses to student formative test scores and administrative data from the three Atlanta schools to examine the following:

• how variation in instructional mode (virtual/hybrid/in-person) and teacher characteristics (such as experience and prior use of technology) affected student achievement
• how the effects varied by gender, race/ethnicity, FRL status, English Learner status, disability status, subject and grade level
• how instructional delivery method (concurrent vs. sequential delivery to in-person and remote learners) and the degree of student online engagement (log-in frequency) during remote learning affected the incidence of student misbehavior once students returned to reopened schools
• how the combination of virtual learning experiences and subsequent recovery strategies affected longer term outcomes such as high-school dropout, high-school graduation and college enrollment.

Products: The research will yield a variety of products that will be tailored to stakeholders' needs and the timelines of decision makers. The researchers will produce research briefs for district and state leaders, which will also be shared with the public, and rapid-response presentations of research findings. The research team will create longer research reports for policymakers, practitioners, and other stakeholders and scholarly work for academic conferences and peer-reviewed journals.

Structured Abstract

Setting: Three Atlanta metro-area school districts that encompass both urban and suburban areas and serve a diverse set of students.

Population/Sample: The sample includes approximately 0.25 million students, including 56 percent who identify as Black, 18 percent Hispanic, 8 percent Asian and 15 percent White.

Factors: The researchers will examine malleable program and policy features including remote learning groupings, teacher assignments to remote/in-person instruction, teacher professional development, in-person and online tutoring programs, extended learning time in core subjects during the school day, supplementary instruction after school, and summer learning opportunities.

Research Design and Methods: For each district recovery strategy employed, the researchers will examine the near-term immediate impacts on student achievement growth, student behavior and student attendance and how the effects varied by student characteristics, subject and grade level. In the first phase of the project, which focuses on school year 2020/21, the researchers will survey parents about family dynamics during the pandemic and at-home learning experiences and combine this information with administrative data from district partners to assess the relationship between instructional mode (in-person/virtual) and both student achievement growth and attendance/engagement. The researchers will analyze a variety of moderating factors, including home environment, student characteristics (e.g., race/ethnicity, gender, English learner status, disability status and household income) and teacher characteristics (e.g., experience, licensure status, educational attainment, and prior use of remote instruction). The researchers will also investigate mediating factors that explain the variation in student outcomes associated with remote learning, such as changes in peer group composition, simultaneous versus sequential instruction of in-person and remote learners, refugee status, and parental preferences for remote learning. The second phase of the project will use data from school years 2021/22 – 2023/24 to analyze the relationships between multiple district recovery/acceleration initiatives and both short-run (student achievement) and long-run (high school completion and college attendance) student outcomes. Sub-group analyzes will also be conducted, with an emphasis on the extent to which programs and policies promote learning and positive long-run outcomes for historically marginalized groups that suffered the largest negative effects from the pandemic.

Comparison Condition: The researchers will use a comparison group to estimate the factors associated with differential success in remote learning by comparing the measured performance of students with similar characteristics and test score histories in the same schools who are exposed to differing levels of remote instruction. The researchers will use program eligibility criteria (for example, low test scores) to create a comparison group for estimating the relationships between recovery strategies and student outcomes. In some cases, the team will compare student achievement and educational attainment for students who just meet the eligibility threshold with those who fall just below the threshold. In other instances, the team will compare outcomes for students who receive the intervention with observationally similar students who do not receive the treatment.

Key Measures: In the first phase the researchers will measure remote learning participation and engagement variables (such as days of remote instruction, attendance and extent of online logins, self-reported emotional and cognitive engagement), student characteristics (such as race/ethnicity, gender, English learner status, refugee status, disability status, household income, ), individual test-score histories, peer groups (such as gender mix and prior behavioral incidents of classmates), teacher characteristics (such as experience, advanced degrees, licensure status, and prior use of online instruction), remote learning delivery (simultaneous with in-person instruction or separate), parental preferences, and student outcome variables (such as nationally normed formative assessment scores, student behavior). In the second phase, in which the team will investigate the efficacy of district recovery strategies, they will incorporate information on program characteristics and eligibility criteria and expand measured outcomes to include high school graduation and college attendance.

Data Analytic Strategy: The researchers will use a variety of cumulative achievement regressions to examine the relationship between remote learning participation and engagement and student test scores as well as mediators and moderators of remote instructional mode effects. To estimate the relationship between district recovery/acceleration strategies and short- and long-term student outcomes, the researchers will employ a mix of analytic strategies, including regression discontinuity, difference-in-differences, and linear regressions with statistical controls for student characteristics.

* This application was submitted to the FY 2022 Education Research Grants (84.305A) competition and awarded in FY 2023 when funds were available.