Inside IES Research

Notes from NCER & NCSER

Measuring In-Person Learning During the Pandemic

Some of the most consequential COVID-19-related decisions for public education were those that modified how much in-person learning students received during the 2020-2021 school year. As part of an IES-funded research project in collaboration with the Virginia Department of Education (VDOE) on COVID’s impact on public education in Virginia, researchers at the University of Virginia (UVA) collected data to determine how much in-person learning students in each grade in each division (what Virginia calls its school districts) were offered over the year. In this guest blog, Erica Sachs, an IES predoctoral fellow at UVA, shares brief insights into this work.

Our Process

COVID-19 has caused uncertainty and disruptions in public education for nearly three years. The purpose of the IES-funded study is to describe how Virginia’s response to COVID-19 may have influenced access to instructional opportunities and equity in student outcomes over multiple time periods. This project is a key source of information for the VDOE and Virginia schools’ recovery efforts. An important first step of this work was to uncover how the decisions divisions made impacted student experiences during the 2020-21 school year. This blog focuses on the processes that were undertaken to identify how much in-person learning students could access.

During 2020-21, students were offered school in three learning modalities: fully remote (no in-person learning), fully in-person (only in-person learning), and hybrid (all students could access some in-person learning). Hybrid learning often occurred when schools split a grade into groups and assigned attendance days to each group. For the purposes of the project, we used the term “attendance rotations” to identify whether and which student group(s) could access in-person school on each day of the week. Each attendance rotation is associated with a learning modality.

Most divisions posted information about learning modality and attendance rotations on their official websites, social media, or board meeting documents. In June and July of 2021, our team painstakingly scoured these sites and collected detailed data on the learning modality and attendance rotations of every grade in every division on every day of the school year. We used these data to create a division-by-grade-by-day dataset.

A More Precise Measure of In-Person Learning

An initial examination of the dataset revealed that the commonly used approach of characterizing student experiences by time in each modality masked potentially important variations in the amount of in-person learning accessible in the hybrid modality. For instance, a division could offer one or four days of in-person learning per week, and both would be considered hybrid. To supplement the modality approach, we created a more precise measure of in-person learning using the existing data on attendance rotations. The new variable counts all in-person learning opportunities across the hybrid and fully in-person modalities, and, therefore, captures the variation obscured in the modality-only approach. To illustrate, when looking only at the time in each modality, just 6.7% of the average student’s school year was in the fully in-person modality. However, using the attendance rotations data revealed that the average student had access to in-person learning for one-third of their school year.

Lessons Learned

One of the biggest lessons I learned working on this project was that we drastically underestimated the scope of the data collection and data management undertaking. I hope that sharing some of the lessons I learned will help others doing similar work.

  • Clearly define terminology and keep records of all decisions with examples in a shared file. It will help prevent confusion and resolve disagreements within the team or with partners. Research on COVID-19 in education was relatively new when we started this work. We encountered two terminology-related issues. First, sources used the same term for different concepts, and second, sources used different terms for the same concept. For instance, the VDOE’s definition of the “in-person modality” required four or more days of access to in-person learning weekly, but our team classified four days of access as hybrid because we define “fully in-person modality” as five days of access to in-person learning weekly. Without agreed-upon definitions, people could categorize the same school week under different modalities. Repeated confusion in discussions necessitated a long meeting to hash out definitions, examples, and non-examples of each term and compile them in an organized file.
  • Retroactively collecting data from documents can be difficult if divisions have removed information from their web pages. We found several sources especially helpful in our data collection, including the Wayback Machine, a digital archive of the internet, to access archived division web pages, school board records, including the agenda, meeting minutes, or presentation materials, and announcements or letters to families via divisions’ Facebook or Twitter accounts.
  • To precisely estimate in-person learning across the year, collect data at the division-by-grade-by-day level. Divisions sometimes changed attendance rotations midweek, and the timing of these changes often differed across grades. Consequently, we found that collecting data at the day level was critical to capture all rotation changes and accurately estimate the amount of in-person learning divisions offered students.

What’s Next?

The research brief summarizing our findings can be downloaded from the EdPolicyWorks website. Our team is currently using the in-person learning data as a key measure of division operations during the reopening year to explore how division operations may have varied depending on division characteristics, such as access to high-speed broadband. Additionally, we will leverage the in-person learning metric to examine COVID’s impact on student and teacher outcomes and assess whether trends differed by the amount of in-person learning divisions offered students.


Erica N. Sachs is an MPP/PhD Student, IES Pre-doctoral Fellow, & Graduate Research Assistant at UVA’s EdPolicyWorks.

This blog was produced by Helyn Kim (Helyn.Kim@ed.gov), Program Officer, NCER.

NCER’s Investments in Education Research Networks to Accelerate Pandemic Recovery Network Lead Spotlight: Dr. Susan Therriault, RESTART Network

We hope you enjoyed yesterday’s network lead spotlight! Today, we would like to introduce Dr. Susan Therriault, director, K–12 Systemic Improvement Portfolio at the American Institute for Research. Dr. Therriault’s network, the PreK-12 Research on Education Strategies to Advance Recovery and Turnaround (RESTART) Network, aims to coordinate activities across research teams and provides national leadership on learning acceleration and recovery from pandemic-induced learning loss, sharing findings from the network with education agencies across the United States. Happy reading!

 

NCER: What are the mission and goals of the PreK–12 RESTART (Research on Education Strategies to Advance Recovery and Turnaround) Network? 

Dr. Therriault: The PreK–12 RESTART Network is an opportunity to develop a coherent and connected research community that speaks directly to the needs of policymakers, leaders, and practitioners. The network focuses on identifying and disseminating evidence-based strategies aligned with the needs of policymakers, leaders, and educators who are serving and supporting accelerated student recovery efforts. This requires the network to identify critical needs of the field and support the research community in developing coherent and coordinated research strategies that build evidence for practices that ensure student recovery—especially among students who have disproportionately struggled in the pandemic context. The PreK–12 RESTART Network will achieve this by need sensing, synthesizing evidence, and building a community that makes meaningful connections between the research community and policymakers, leaders, and educators.

NCER: Why is the PreK–12 RESTART Network important to you? 

Dr. Therriault: The PreK–12 RESTART Network is important to me because, as a researcher, I have watched how the pandemic and subsequent aftershocks of the pandemic have created disruptions to our lives and our public pre-K to 12 education system. The pandemic created fragmentation and division as communities responded and supported individuals in a context marked by social distance and separation. While there are many common challenges across communities, limited social connection affected our ability to share evidence-based solutions and to equitably address the needs of all members of our communities, especially those communities most adversely impacted by COVID-19, including Black and Latinx communities and those marked by poverty and housing and food insecurity.

NCER: How do you think the PreK–12 RESTART Network will impact the pre-K to 12 community?

Dr. Therriault: The PreK–12 RESTART Network is an opportunity to develop a coherent and connected research community that speaks directly to the needs of policymakers, leaders, and practitioners. The key differentiator of the network is that it is purposefully designed to assess needs and engage the research community in providing insight and building evidence for solutions to address those needs.

The network has an important role to play in drawing researchers together to develop measurement solutions and build consensus for approaches to conducting and making meaning of research so that it informs the field. These solutions will be shared with the field to create a more coherent research agenda informed by needs.

The network will ensure that policymakers, leaders, and educators are able to easily access network evidence syntheses and research-team findings through multiple communication formats. Information will be shared through actionable guidance and recommendations purposefully designed for these audiences. A combination of strategies will ensure accessibility. These include digital tools and dashboards that school leaders can easily use to adapt to their circumstances and access to evidence-based strategies by offering recorded webinars, tutorials, videos, and other learning formats.

NCER: What’s one thing you wish more people knew about recovery in pre-K to 12 education? 

Dr. Therriault: The magnitude of the challenge of pandemic recovery in the pre-K to 12 education system is vast and will require new ways of approaching education and support for students. In turn, with the investment of American Rescue Plan funds in schools, this will likely lead to evidence-based innovation and deeper understanding of how to design an education system, district, and school to meet student needs.

The needs of students are varied and highly connected to the experience during the pandemic and after the pandemic; thus, family and community factors are highly relevant and critical to understanding student needs and strategies to address these needs. Recently released NAEP scores provide evidence of the variation and suggest more-significant losses in mathematics and English language arts for students living in low-income households compared to their peers who are not. Further, students living in low-income households were more likely to report not having a place to do work or access to a computer or adequate uninterrupted time compared to their peers. These are critical factors in a remote and even a hybrid learning environment. These differences in experience exacerbate differences in outcomes during the pandemic.

NCER: What are some of the biggest challenges to recovery in pre-K to 12 education? 

Dr. Therriault: The amount of need among students and their families and the fatigued pre-K to 12 education system workforce are the biggest challenges to recovery. Adding to this challenge is the focus on expanding learning time through summer school or longer school days in an effort to accelerate learning. This requires teachers and leaders at a time when, like many of us, they are experiencing burnout.

Finally, we know that most students suffered learning loss and more during the pandemic. Supporting students emotionally as well as academically is necessary for recovery. While many schools have provided emotional support to students prior to the pandemic, the current need is far greater than schools have experienced. This will require extensive outreach to community support and services and additional interventions.

NCER: What are some effective ways to translate education research into practice so that your work will have a direct impact on states, districts, and schools? 

Dr. Therriault: There are several ways we plan to support the translation of research to practice through the PreK12 RESTART Network. These include:

  • Understanding and sharing needs of the field.
    • Conducting needs assessments of the field to examine the evolving needs and share these with the research community.
  • Identifying, sharing, and amplifying evidence-based strategies that respond to the needs of the field.
    • Exploring existing and emerging research to identify evidence-based strategies and interventions that align evidence syntheses with the needs of the field.
    • Providing actionable guidance and recommendations that can be easily implemented across different schools and are customized based on need.
    • Creating digital tools that school leaders and educators can use to adapt to their circumstance.
  • Building a coherent and coordinated research community focused on pandemic recovery research.
    • Connecting and building consensus among researchers through convenings and solutions working groups that address challenges to conducting research in the pandemic context.
    • Empowering research teams to build studies that align with needs in the field through meaningful connections with policymakers, leaders, educators, and other members of the research community.
    • Supporting engagement and preparation of early-career researchers through trainings and networking opportunities.

NCER: What are some barriers to the uptake of the research outcomes by these organizations?

Dr. Therriault: One of the critical barriers to uptake is timing. States, districts, and schools cannot wait for findings and results—they must act to support the students they have in their classrooms right now. The syntheses and reviews of prior research will help point educators in the direction of interventions and other supports that have a strong evidence base. The researchers participating in the RESTART Network will be supported in rapidly sharing and disseminating findings over the course of their studies to inform decision-making about how best to help students’ academic recovery.


Thank you for reading our conversation with Dr. Susan Therriault! We hope you’ve enjoyed getting to know NCER’s network leads throughout our grantee spotlight series. Let us know your thoughts on the series on Twitter at @IESResearch.

Save the Date: Leveraging Evidence to Accelerate Recovery Nationwide (LEARN) Network Launch Event

Join us on January 19, 2023, from 3pm EST-4:30pm EST, as members of the IES-funded LEARN (Leveraging Evidence to Accelerate Recovery Nationwide) Network convene publicly for the first time to share their network's goals and vision. Learn more from the network teams during this virtual event

and hear from IES Director Mark Schneider about his hopes for the LEARN Network in the coming years as IES looks to the future with a focus on progress, purpose, and performance.

The LEARN Network was established to focus on adapting and preparing to scale existing, evidence-based products to address learning acceleration and recovery for students in K-12, particularly for students from underrepresented groups disproportionately affected by the COVID-19 pandemic. In addition to generating solutions to the nation’s most pressing challenges to COVID-19 recovery within the education sector, IES expects that the combined efforts of this network will lead to the establishment of best practices for how to prepare to effectively scale evidence-based products.

The LEARN Network includes a scaling lead and four product teams. The scaling lead, led by a team at SRI International, is facilitating training, coaching, and collaboration activities with product teams; ensuring educator needs and perspectives are addressed; and providing a model for the field that ensures evidence-based products are developed with the potential to achieve impact at scale for students—particularly those in most need—from the start. Product teams are focused on preparing to scale literacy products for students in K-3 (Targeted Reading Instruction; Grantee: University of Florida), 4th-5th grade (Peer-Assisted Learning Strategies; Grantee: AIR), and middle school (Strategic Adolescent Reading Intervention; Grantee: SERP) as well as a math product for students in 5th grade (Classwide Fraction Intervention combined with Math Peer-Assisted Learning Strategies; Grantee: AIR).

Registration is now open, and we hope to see you there! For more information on the event and to register, visit https://learntoscale.org/

Graduate Student Reflections on Engaging Research Opportunities

Engaging students in research can enrich their knowledge and support their future confidence to pursue research careers. In this interview blog, Dr. Allen Ruby, Associate Commissioner for the Policy and Systems Division at NCER, asked four doctoral graduate students at Montclair State University, Melissa Escobar, Taylor Walls, Hannah Thomas, and Marline Francois, to reflect on what attracted them to an IES-funded research project led by Dr. Carrie Masia. The project aims to improve education outcomes for Black American high school students with anxiety attending urban public schools through the development of culturally-responsive interventions.

What are your research interests, and how does this project align with your interests?

Melissa Escobar (ME): My research interests focus on optimizing access to evidence-based treatments for racial minority youth by training frontline providers in community and primary care settings to deliver them. I distinctly remember when I decided to pursue a career in psychology. I was working at a community youth center when the struggles of a Latinx mother deeply impacted me. Her husband's deportation to Mexico significantly altered her son's mood and schoolwork. She tearfully confided in me about her difficulties accessing mental health services for her son. She struggled to find a qualified provider who she believed understood their family's concerns, and the high cost of services and transportation prevented her from seeking care. Seeing the combination of cultural and structural barriers that influence mental health disparities within marginalized groups, I now align myself with research that advocates for high-quality depression and anxiety treatments in accessible locations for minority youth. This is why I found Dr. Masia's project a perfect fit for me. The project links the behavioral health and education sector to improve the mental health and academic achievement of historically marginalized youth with impairing anxiety.

Taylor Walls (TW): My research interests center around developing, implementing, and evaluating culturally sensitive interventions for children and adolescents with internalizing disorders in schools. A primary goal of this project is to use a school-based group intervention that has been shown to be effective in reducing social anxiety and revise it to address the unique needs of Black American students. It aims to consider the context of urban public schools and the culture of Black American adolescents. I have read about the importance of cultural adaptations to improve the quality and availability of these interventions for racial and ethnic minorities, and I welcome the opportunity to work closely on a project like this firsthand.

 

Hannah Thomas (HT): My research interests include optimizing evidence-based interventions for children and adolescents and the role of risk and protective factors in the development of internalizing disorders. My interest in these areas began when I worked at a summer program focused on bringing high school-aged student athletes, often from underserved communities, to learn leadership and sport psychology skills. This was a transformative experience, ultimately solidifying my interest to work with youth and interventions that teach skills to handle adversity. I was drawn to this IES project because it provides the opportunity to work with youth in a meaningful and impactful way.

 

Marline Francois (MF): My research interest is in exploring the psychological well-being of adolescent Black girls that experience racial and gender discrimination in education spaces. Furthermore, I am interested in creating gender and race-specific interventions for Black adolescent girls. My interest began after spending 15 years working as a therapist and recognizing the lack of interventions specifically for Black youth. I was also affected by the lack of adequate mental health services for Black youth coupled with their alarming increase in suicide rates. What I love about this project is that it aligns with my research interests and directly involves adolescents, which provides opportunities to learn from them and have them share their lived experiences.

 

What has been the most exciting aspect of this project for you?

ME: The most attractive part has been the opportunity to hear directly from students and parents on how to best address the needs of racial-ethnic minority families and develop culturally sensitive assessments and treatment strategies. Fully engaging with the community makes the work more meaningful to me. Also, the chance to have a hands-on approach in the research process from collecting qualitative data to developing interview guides and coding schemes makes me feel like my contributions are making a difference.

TW: As a fourth-year graduate student, this project is particularly exciting for me because I will be analyzing a portion of the data for my dissertation. It has expanded my research skills to formulate my own research questions to contribute information that is novel and of interest to my field. Furthermore, I enjoyed the aspects of this project that mirror my work as a clinician — speaking with children and their parents in one-on-one and small group settings to hear about their experiences and their feedback on how this program may be better tailored to their community. Having conversations with the individuals we want to impact makes this work particularly meaningful.

HT: The most exciting aspect of this project is the ability to be involved in multiple roles. I started at the beginning of the summer, and so far, I have been involved in conducting focus groups and developing a coding scheme for interview transcriptions. This excites me because I am able to diversify my skills as a researcher and gain experience in various research methods that may be useful for my dissertation.

MF: As a 3rd-year doctoral candidate, it has been exciting to see the process of this project from the beginning and being able to interview and interact with students and parents. I enjoyed the recruitment process and conducting focus groups and individual interviews. As a qualitative researcher, I appreciate the hands-on experience of learning how to conduct and the in-depth experiences shared by the students and parents, which will aid us in creating a more culturally sensitive intervention for Black youth.

What do you look for in a research supervisor or mentor?

ME: In addition to similar research interests, I look for a mentor who is respectful of personal boundaries. As graduate students, we have a lot of different responsibilities and having a mentor that shows you how to balance those roles is vital in keeping students engaged and successful. I have been fortunate to have a mentor who ensures that all work is evenly spread across all research team members. I also appreciate mentors who considers their students' personal goals and finds opportunities that align with them. For example, at the beginning of my graduate program, my mentor asked me what my goals were for the year, the program, and what type of work setting I saw myself in after graduation. This conversation has been beneficial in finding research opportunities, grants/scholarships, and clinical experiences that will help me meet my goals outside of working on research. Lastly, I take into consideration how available the mentor is. It is essential to maintain good communication through regular meetings and/or emails. This way, regular communication and feedback can happen in appropriate time frames, and any issues that arise are resolved quickly.

TW: It's important to me that a mentor has both interest in and time to support me in fostering my competence as a research professional and advancing my short- and long-term goals. This can be demonstrated in a number of ways, including recommending academic and professional development activities that will build my skills, providing constructive feedback in a respectful and supportive manner, and helping me manage challenges as they arise. I also prefer that my mentor shares in my passion for child-focused research and is eager to connect me with collaborators for projects or networking. Over the years, I've learned that I work best with mentors who grant me autonomy in my work, but I also benefit from frequent check-ins and strict deadlines. Finally, I appreciate a research mentor that provides encouragement and flexibility and acknowledges the importance of self-care and well-being.

HT: I looked for a research supervisor/mentor whose research interests aligned with mine. I found this to be important because not only are they knowledgeable in the area of research that I am interested in, but they also provide the right research opportunities to develop me on the road to an independent career.

MF: It was important for me to find a mentor with a research background that aligns to my research interests. I wanted to have someone that cared about my research interests and professional growth. Furthermore, it was important for me to find someone who was not afraid to give me constructive criticism on my ideas but also assist me with strategically planning for my future. As someone who values balance, it is also important for me to have a mentor who values my well-being.

What challenges have you faced when trying to find research projects that appeal to you, and what feedback would you give to graduate programs or faculty to better engage students in research? 

ME:  As a first-generation student, I did not know how to navigate finding research opportunities or emailing professors about potential opportunities. I quickly learned that most research opportunities aren't advertised and finding a role on a research team usually comes from word of mouth. I would encourage programs and faculty to do more to advertise research opportunities. I would also recommend that faculty welcome the involvement of undergraduate students in their labs. My research career began during the sophomore year of my undergraduate education. I would not have the experience I have today if that professor had not given me a chance. Even if the roles are small like doing audio transcription or data entry, all experience is valuable. Another suggestion would be to create a mentorship model within the lab with more senior students mentoring newer students. When relationships like this are built, students may feel more comfortable to try out new roles in the lab.

TW: In order to find research projects that appeal to me, I am diligent about seeking out faculty who are already doing work that interests me and are open to bringing on a collaborator. I appreciated that my graduate master's program hosted an open house at the beginning of each semester where students could meet faculty and be oriented to the research labs that were available to them. What helped me first want to get engaged in research was meeting faculty who were outwardly passionate about their work and created unique avenues for their students to get involved.

HT: I feel that I have been fortunate in obtaining research projects that appeal to me, and I attribute that to aligning myself with the right mentors. My past and current mentors are collaborative with other faculty and labs, which have allowed me to participate in a variety of research opportunities, further refining my research interests.  My suggestion for graduate programs and faculty to better engage students in research is to encourage collaboration across faculty and labs. 

MF: When I initially started my doctoral program, it was challenging to find the right research project that aligned with my interests. However, I believe it is important as a doctoral student to be flexible and adaptable to learning from other mentors. Furthermore, I found myself connecting with scholars and students from other disciplines and universities that had similar interests as myself. This has allowed me to learn to properly advocate for my needs and find a research lab that was more suitable for me. The advice I would give to graduate programs is to create more opportunities for graduate students across departments and disciplines tailored to various research needs. I also believe that universities can create more professional development training opportunities to better engage students in research, such as how to apply for competitive fellowships and grant opportunities during their studies.  


This blog was produced by Allen Ruby (Allen.Ruby@ed.gov), Associate Commissioner for Policy and Systems Division, NCER. 

 

Six Strategies for Effectively Communicating Research Findings to Decision Makers

Researchers must possess the ability to clearly communicate research findings to non-technical audiences, including decision makers who may have limited time and varying levels of tolerance for data-rich reports. We and our colleagues recently honed these skills while preparing research briefs for the Virginia Department of Education (VDOE) as part of an IES-funded partnership project between VDOE and the University of Virginia exploring the impacts of the COVID-19 pandemic on students and teachers. These briefs are a key mechanism through which our project services the purpose of IES’s Using Longitudinal Data to Support State Education Policymaking Grantmaking Programs to generate useful findings to inform the decision making of policy makers and education leaders at the state, district, and school levels.

In their initial feedback, VDOE described the briefs as “too technical.” When we led with the numbers, our intended audience quickly became overwhelmed by the need to also interpret the findings on their own. Our conversations with VDOE provided helpful direction on how we could revise the briefs to better reach non-technical, decision-making audiences in Virginia and beyond. We share six strategies we have applied to all our research briefs.

  • Yes, briefs need a summary too: The draft briefs were short (4-7 pages) inclusive of figures and endnotes, and they began with a list of key findings. Based on the feedback, we morphed this list into a proper summary of the brief. Many of the decision makers we want to reach only have time to read a page summary, and that summary needs to be self-contained. Without additional context, the initial list of key findings would have had minimal impact.
  • Lead with the headline: Numbers are a powerful tool for storytelling; however, too many numbers can also be hard for many people—researchers and non-researchers alike—to consume. We therefore edited each paragraph to lead with a numbers-free sentence that provides the main take away from the analysis and followed that up with the supporting evidence (the numbers).
  • Answer the question: Our initial groundwork to develop solid relationships with agency staff allowed us to identify priority questions on which to focus the briefs. While several tangential but interesting findings also resulted from our analysis, the briefs we developed only focused on answering the priority research questions. Tangential findings can be explored in more depth in future research projects.
  • Accurate but not over-caveated: All research makes some assumptions and has some limitations. The average non-technical audience member is unlikely to want a thorough detailing of each of these; however, some are too important to exclude. We chose to include those that were most vital to helping the reader make the correct interpretation.
  • A picture speaks a thousand words: This was something at which our initial drafts succeeded. Rather than providing tables of statistics, we included simple, well-labeled figures that clearly presented the key findings graphically to visually tell the story.
  • Conclude by summarizing not extrapolating: The purpose of these briefs was to describe the changes that the pandemic wrought to Virginia’s public schools and convey that knowledge to decision makers charged with plotting a course forward. The briefs were not intended to provide explicit guidance or recommendations to those decision makers.

These strategies, of course, are also useful when writing for technical audiences. While their training and experiences may equip them to consume research that doesn’t exhibit these six strategies, using these strategies will enhance the impact of your research findings with even the most technical of audiences.


Luke C. Miller is a Research Associate Professor at the University of Virginia’s School of Education and Human Development. He is the lead researcher and co-Principal Investigator on the IES-funded project led by VDOE in partnership with UVA.

Jennifer Piver-Renna is the Director of the Office of Research in the Department of Data, Research and Technology at the Virginia Department of Education. She is the state education agency (SEA) co-Investigator on the IES-funded project.

This blog was produced by Allen Ruby (Allen.Ruby@ed.gov), Associate Commissioner for Policy and Systems Division, NCER.