|Evaluation of the Learning Renewal – Social Emotional Learning Programs for Supporting Pandemic Recovery with Illinois State Board of Education
|American Institutes for Research (AIR)
|Prekindergarten Through Grade 12 Recovery Research Network [Program Details]
|3 years (09/01/22 – 08/31/25)
|Exploration and Efficacy
Co-Principal Investigators: LiCalsi, Christina; Dixon, Brenda M. Dixon
Education Agency: Illinois State Board of Education (ISBE)
Related Network Teams: This research project is part of the Prekindergarten Through Grade 12 Recovery Research Network, which includes the following other projects — PreK-12 RESTART Network Lead (Research on Education Strategies to Advance Recovery and Turnaround) (R305X220009), A Researcher-Practitioner Partnership to Improve Pandemic Recovery Efforts in California (R305X220028), Mitigating the Impact of the COVID-19 Pandemic Through Curriculum-Based Approaches to Learning Acceleration in Grades K–2 in Chicago (R305X220033), Assessing the Long-Term Impacts of School Extension Programs on Student Re-Engagement and Learning Recovery (R305S220003), Lessons from the Pandemic: The Effects of Remote Instructional Delivery and Recovery Strategies on Student Outcomes (R305A230400)
Purpose: This network study has two goals: (a) to generate evidence about the effects of the Learning Renewal–Social Emotional Learning Programs (LR-SEL), as part of Illinois's efforts to support recovery from the impact of the COVID-19 pandemic, and (b) to generate useful feedback about effective strategies for implementing and scaling social and emotion-focused COVID-19 recovery programs for schools with high proportions of students experiencing economic disadvantage. Findings from the study are expected to help ISBE better understand the effects, implementation, and costs of the LR-SEL programs and improve the agency's ongoing SEL supports for districts and schools.
Project Activities: The project has three core components: an outcome evaluation, an implementation evaluation, and a cost study. In addition, the research team will conduct a variety of activities to disseminate the findings to various LR-SEL stakeholders and other stakeholders who may benefit from the findings of the study. The team will collect student and teacher administrative data, administer student and teacher surveys, and conduct interviews with key personnel at Regional Offices of Education (ROEs), Intermediate Service Centers (ISCs), SEL Hubs, and participating districts and schools. Data collection will occur during all 3 years of the project, with ongoing analysis, reporting, and dissemination across all 3 years. Opportunities will be made available for early career scholars to work on all aspects of the project.
Products: This research team will produce an interactive dashboard or tracking tool through which the state can view the types of activities and implementation progress. The research team will present project results back to the communities in which data were collected via the SEL Hubs, as well as through webinars and regional meetings for other audiences across the state. In addition, the research team will produce a publicly available final report and peer-reviewed journal articles, as well as present at academic and practitioner conferences. The research team also will disseminate findings to a broad audience through user-friendly infographics and social media postings.
Setting: The evaluation will include Illinois schools that participated or are currently participating in the LR-SEL programs between 2021–22 and 2024–25 and selected comparison schools. The schools will be located across the state.
Population/Sample: The sample for the outcome analysis of REACH will include 1,038 REACH schools for 1-year effects, 609 REACH schools for 2-year effects, 179 schools for 3-year effects, with an equal number of matched comparison schools in each analysis. Assuming at least one school from each of the 136 community partnership grants awarded in spring 2022, the sample for the outcome analysis of community partnership grants will include 136 community partnership grant schools for both Year 1 and Year 2 effects, and an equal number of matched comparison schools. The implementation evaluation will include interviews with staff members from all 35 of Illinois's ROEs, three ISCs, six school districts, and 12 schools within those districts that are participating in the LR-SEL programming.
Factors/Intervention: The project team will examine three SEL activities (REACH Expansion, SEL Hubs, and the Community Partnership Grants) that are part of ISBE's efforts to support recovery from the effects of the COVID-19 pandemic. The team will examine the programs' effect on student and teacher behavior, student academic learning, school climate, and changes in student and teacher social-emotional wellness. In addition, the study will examine factors that facilitate or impede the implementation of these programs and the cost to schools and districts for implementing the programs.
Research Design and Methods: Each component of LR -SEL will be evaluated separately. The research team will use a quasi-experimental design (QED) with matching to construct similar groups of REACH intervention schools and comparison schools, and community partnership grant schools and comparison schools. After the propensity scores have been generated, the team will match schools within SEL Hubs (i.e., geographic areas). Exploratory analyses will examine whether the LR-SEL program components lead to improvement in student and social-emotional outcomes. The implementation evaluation will use a tiered, purposive sampling approach and a series of interviews and case studies to examine meaningful variation in Learning Renewal-SEL implementation across schools, districts, ROEs, and area SEL Hubs.
Control Condition: For each analysis in the QED evaluation, a program school will be matched with a school in the same Hub (i.e., geographic area) that did not implement either REACH or community partnership grants but are similar to the treatment schools in key characteristics such as student composition, prior achievement, and prior school climate.
Key Measures: Student behavioral measures include attendance and discipline events. Teacher behavioral measures include attendance and retention. Student academic learning measures include performance on state standardized English language arts and mathematics exams, on-track status, and graduation. Student and teacher social-emotional wellness, mental health, and school climate will be assessed using reliable and validated survey-based measures.
Data Analytic Strategy: The QED analysis will use a mixed effects two-level hierarchical linear model with students nested in school pairs using the matched sample. Differential effects analysis will be conducted to assess the extent to which the effect of the LR-SEL program is moderated by student, teacher, and school characteristics. In addition, mediation analysis will be conducted to explore whether the effects of the program on student outcomes are mediated by school climate. Descriptive and correlational analysis will be used to examine how student and teacher social-emotional outcomes change over time and identify factors that are associated with the outcomes. The qualitative analysis will use NVivo qualitative software to analyze interviewees' responses by mapping them onto a coding structure that aligns with the topics covered in the semi-structured interview protocols.
Cost Analysis: The cost analysis will include a random sample of 10 schools that implement or participate in each of the three LR-SEL programs (REACH Expansion, SEL Hubs, and the Community Partnership Grants) and will use the ingredients approach to identify the costs associated with all resources used to implement the LR-SEL programs and calculate the overall and per-student cost of the resources used in the implementation of each LR-SEL program.