Skip Navigation

Home Blogs New study estimates student learning changes following school building closures due to the COVID-19 pandemic

New study estimates student learning changes following school building closures due to the COVID-19 pandemic

Midwest | December 01, 2021

Study estimates learning changes due to the pandemic

The COVID-19 pandemic continues to disrupt student learning across the country. In Illinois, all public and nonpublic school buildings were closed by executive order in March 2020 to protect the health and safety of students and educators during the pandemic. Districts across the state shifted to remote instruction for the rest of the 2019/20 school year, and many continued with distance education, or a hybrid of distance and in-person learning, in the 2020/21 school year.

To understand the relationship between these school building closures and the suspension of in-person instruction, Illinois state and district leaders sought research and evidence to inform decision making related to providing targeted or differentiated supports to students. Regional Educational Laboratory (REL) Midwest partnered with the Illinois State Board of Education to conduct a study that examines student learning changes that occurred as schools shifted to remote learning during the pandemic.

>> Read and download the full report and related infographic.

About the study

The Illinois State Board of Education requested this study to understand the nature of the learning changes that occurred in the state during the pandemic. The study team explored how student learning in mathematics and reading changed as of fall 2020 following extended school building closures due to the COVID-19 pandemic. The study also examined how changes to student learning varied by student characteristics and district size.

To conduct the study, the Illinois State Board of Education invited all Illinois districts serving students in grades 3–8 to participate by sharing their interim assessment data from fall 2020 and multiple prior years. The state agency collected data from 17 Illinois districts. The REL Midwest team then analyzed the data to measure how student learning had changed in fall 2020 relative to fall terms prior to the pandemic. The study examined a combination of fall Northwest Evaluation Association Measures of Academic Progress (NWEA MAP) data provided by the participating districts as well as student, school, and district records provided by the Illinois State Board of Education. Several additional districts were interested in the study but did not have sufficient data at the time to support participation. REL Midwest provided training to these districts to help them understand strategies they could implement on their own to assess the learning changes their students had experienced and to support their use of this information in conversations and planning.

What did the study find?

Key findings include the following:

  • In fall 2020, following the onset of the COVID-19 pandemic, students scored lower than expected in mathematics, but not in reading. Student performance on the NWEA MAP mathematics assessment was lower than expected in grades 4–8, compared to how similar students scored in earlier years. The estimated amount of unfinished learning was largest in grades 6, 7, and 8.
  • Student characteristics, such as English learner status and special education status, were associated with changes in learning in mathematics, but whether these student groups experienced more or less unfinished learning varied by grade level. In grades 4 and 5, for example, students who were eligible for the national school lunch program experienced more unfinished learning in mathematics than students who were not eligible. In grades 7 and 8, students in special education had more unfinished learning in mathematics than students who were not in special education. However, in grades 4, 5, and 6, students in special education had less unfinished learning than students who were not in special education. In grades 6 and 8, English learner students had more unfinished learning in mathematics than students who were not English learner students, and in grade 4, English learner students had less unfinished learning than students who were not English learner students. Finally, in grades 4 and 6–8, White students had more unfinished learning in mathematics than students of another race or multiple races.
  • District size was not related to changes in learning in mathematics. The study compared changes in student learning between smaller and larger districts, with larger districts defined as those with total enrollment of 4,000 students or more. District size was not related to changes in learning in mathematics in any of the grades studied.

These findings were generated after adjusting for other factors, including students’ eligibility for the national school lunch program, race/ethnicity, English learner student status, special education status, and prior academic achievement, as well as a time trend and other factors.

In addition, the study had several limitations that should be considered when reviewing the report findings. First, because all students in Illinois experienced the effects of the pandemic at the same time, there is no comparison group against which to measure academic outcomes, and the study should not be viewed as causal. Second, changes with the fall NWEA MAP assessment may have affected the findings. Prior to the pandemic, the NWEA MAP was administered to all students in person. In fall 2020, many districts administered the assessment to students remotely. Third, the study sample only includes four comparison time points prior to the onset of the COVID-19 pandemic. Thus, if student test scores tended to fluctuate from year to year, the study may not have estimated changes in learning precisely. Finally, the study districts, and the students they serve, differ from the overall population of districts in Illinois in multiple ways. For instance, none of the districts in the study sample are in the southern regions of the state, and the sample does not include Chicago Public Schools, which enrolls 18 percent of all public school students in Illinois. For more information, see the full report.

What can we take away from the findings?

The study’s findings can inform the strategies implemented by Illinois education leaders as they allocate resources and provide supports for students as they return to full-time in-person learning.

School and district leaders may want to identify and implement evidence-based strategies and interventions that can accelerate learning in mathematics. In five of the six grade levels analyzed, students scored worse than expected in mathematics following the onset of the pandemic, after adjusting for other factors.

Additional research is needed to further inform districts’ and schools’ efforts, as well as state-level efforts, to support students through the duration of the pandemic and beyond. District and school administrators working to allocate resources and support students with learning recovery may want to gather additional information that can help identify why student learning changed as it did. Administrators may want to focus these efforts on student groups that experienced the most unfinished learning and identify the drivers of the learning changes. Education leaders can then use the information to identify and provide appropriate assistance to support the targeted groups’ learning recovery.

Research may also explore changes in student learning for a more comprehensive and representative sample of students. This study used data from a small sample of districts, whose students differ from the Illinois population in important ways. Future research could explore opportunities for gathering and analyzing data from a larger sample of districts to produce findings that better represent the population of students in Illinois.

Finally, additional research is needed explore changes in student learning over a longer period of time. This study focused on changes in student learning as of fall 2020, after approximately six months of disruption to school and life due to the COVID-19 pandemic. However, the pandemic has persisted well beyond this time, and many districts continued to serve students in remote or hybrid formats throughout the 2020/21 school year.

"The study is still relevant and has the same sense of urgency today as it did one year ago," said Brenda Dixon, Ed.D., research and evaluation officer at the Illinois State Board of Education. "As long as the pandemic exists and continues to have an impact on students and on their learning, this study will still be relevant and important to understand. This study allows districts to know more about where to allocate their dollars. According to this study’s findings, there were larger impacts on student learning in math. We even learned what grades were most affected, which gives districts information to target their efforts with strategies such as afterschool tutoring and other interventions they’re trying to put in place to accelerate learning going forward."

Related resources

For more information about the study findings, read the full report and related infographic. Watch this short video to hear from the report authors and district leaders about the study findings and implications for Illinois states and districts.

To learn more about how REL Midwest is supporting the work of districts and states during the pandemic, see the following related resources:

  • REL Midwest partnered with the Ohio Department of Education to create a toolkit and training series that states and districts can use to assess learning changes through qualitative and quantitative approaches. In addition, REL Midwest used the toolkit in a training for Illinois districts that were not able to participate in the study follow-up training.
  • To help education leaders respond to the COVID-19 pandemic, experts from REL Midwest produced blog posts that summarize strategies for using educational technology and teaching young children, English learner students, and students with disabilities in a remote environment. In addition, REL Midwest developed a set of infographics on literacy and math instruction for the Michigan Department of Education based on the What Works Clearinghouse Teaching Math to Young Children and Foundational Skills to Support Reading practice guides.
  • REL Midwest also hosted a series of webinars focused on remote learning and the needs of stakeholders during the pandemic. The first webinar, held in March 2020, discussed evidence-based practices for bringing in-person classroom strategies to a virtual learning environment. The follow-up webinar, held in April 2020, provided more detail on the topics identified by regional stakeholders, including social and emotional learning, adult learning, and online learning platforms.
  • The REL Program COVID-19 page highlights additional evidence-based resources developed by the network of RELs in response to the COVID-19 pandemic.

Author(s)

Maggi Ibis

Maggi Ibis

Connect with REL Midwest