IES Blog

Institute of Education Sciences

Conducting Education Research During Covid-19

Since the start of the pandemic, we have all heard about the unprecedented changes to schooling in the U.S. and the ways that educators, students, and families have been adapting to the new reality.

Education researchers have also been adapting their work due to school closings, canceled testing, and different school reopening plans in the 2020-21 school year.

How have education researchers handled the new reality?

Some researchers have been busy compiling and disseminating research findings to support districts and schools to continue instruction during the pandemic. For example, evidence-based recommendations were made available to help parents and schools pivot to a virtual environment (from very young children up to the postsecondary level), maintain engagement, address mental health (including in rural areas), protect against learning loss, and decide how to prioritize needs when considering re-opening. And many education technology researchers and developers have provided online resources to schools.

Other researchers have been working hard to understand the overall disruption to schooling due to COVID-19 and the ramifications on student learning around the world.  For example, there have been efforts to keep track of school closures, document what is happening in schools across the country (including in rural districts), study the switch to online learning and attend to unequal access to technology for remote learning, forecast funding scenarios, and examine changes in teacher recruitment.

In addition, education researchers are thinking about new ways to conduct research in light of the changes to schooling. They are looking at alternatives to standardized testing, new approaches to teaching and learning to strengthen schools moving forward, and ways to rebuild our education systems after the pandemic. Indeed, there are myriad ways that education researchers can and are using their skills to continue to support education during this unprecedented time.

How has COVID-19 impacted IES-funded education research studies?

IES realizes that the pandemic has changed things in ways that may complicate education research – both how it is conducted and how it is interpreted. So, we are actively working with grantees to help ensure the integrity of their work and to respond to the needs, interests, and concerns of the schools and colleges they are working with and the communities they are trying to help. In a follow-up to an IES-funded study on students in foster care, a researcher-practitioner partnership in Colorado is examining the implications of challenging circumstances such as COVID-19 on the postsecondary education of vulnerable youth.

Many IES-funded researchers have had to alter their research plans to accommodate the needs of their partner schools and overcome the challenges posed by the abrupt transition to virtual learning. Because of continued uncertainty, they may need to change plans again. Program officers at IES have been working with grantees on a case-by-case basis to adapt their timelines and, in some cases, their research designs.

IES’s priority is to help researchers maintain scientific rigor while holding a realistic view of what can and cannot be done this year. As we work with our grantees, we take into consideration where the project is in its overall timeline. For example, if the project has collected all of its data and is in the final analysis stage, the remaining work may not be affected. Or, if a project has not yet started to begin an intervention in schools, it can pause during the 2020-2021 academic school year and resume in 2021-2022. Still, other projects may find themselves unable to either continue or pause. These projects may not be able to achieve their initial purpose and may need to end.

Despite some of the challenges, the pandemic offers a unique natural experiment for learning and instruction, as well as opportunities for innovation that can ultimately benefit education. IES, our funded researchers, and the communities that rely on research evidence continue to pull in the same direction: building strong evidence to inform policy and practice. Through collaboration and dialog, we will work together to ensure that data and results are meaningful, valid, and as timely as possible. IES will continue to focus on high-quality education research to improve student learning and achievement both now and in the future.

Stay tuned for future blog posts on what our researchers are doing to address some of the challenges that face educators, families, and policymakers during this unprecedented time!


Written by Corinne Alfeld (Corinne.Alfeld@ed.gov), National Center for Education Research.  

What National and International Assessments Can Tell Us About Technology in Students’ Learning: Eighth-Graders’ Experience with Technology

The use of technology has become an integral part of life at work, at school, and at home throughout the 21st century and, in particular, during the coronavirus pandemic.

In this post, the second in a three-part series, we present results from the NAEP TEL and ICILS student questionnaires about students’ experience and confidence using technology (see the first post for more information about these assessments and their results). These results can help to inform education systems that are implementing remote learning activities this school year.

Uses of information and communication technologies (ICT) for school

Both NAEP TEL and ICILS collected data in 2018 on U.S. eighth-grade students’ uses of ICT in school or for school-related purposes.

According to the NAEP TEL questionnaire results, about one-third of U.S. eighth-grade students reported that they used ICT regularly (i.e., at least once a week) to create, edit, or organize digital media (figure 1). About a quarter used ICT regularly to create presentations, and 18 percent used ICT regularly to create spreadsheets.



According to the ICILS questionnaire results, 72 percent of U.S. eighth-grade students reported that they regularly used the Internet to do research, and 56 percent regularly used ICT to complete worksheets or exercises (figure 2). Forty percent of eighth-grade students regularly used ICT to organize their time and work. One-third regularly used software or applications to learn skills or a subject, and 30 percent regularly used ICT to work online with other students.



Confidence in using ICT

Both the 2018 NAEP TEL and ICILS questionnaires asked U.S. eighth-grade students about their confidence in their ICT skills. NAEP TEL found that about three-quarters of eighth-grade students reported that they were confident that they could—that is, they reported that they “probably can” or “definitely can”—compare products using the Internet or create presentations with sound, pictures, or video (figure 3). Seventy percent were confident that they could organize information into a chart, graph, or spreadsheet.



ICILS found that 86 percent of U.S. eighth-grade students reported that they knew how to search for and find relevant information for a school project on the Internet (figure 4). Eighty-three percent knew how to both upload text, images, or video to an online profile and install a program or app. About three-quarters of eighth-grade students knew how to change the settings on their devices, and 65 percent knew how to edit digital photographs or other graphic images.



Years of experience using computers

In the 2018 ICILS questionnaire, U.S. eighth-grade students were also asked how many years they had been using desktop or laptop computers. One-third of eighth-grade students reported using computers for 7 years or more—that is, they had been using computers since first grade (figure 5). This finding was similar to results from the Computer Access and Familiarity Study (CAFS), which was conducted as part of the 2015 NAEP. The CAFS found that in 2015, about 35 percent of eighth-grade public school students reported first using a laptop or desktop computer in kindergarten or before kindergarten.

Nineteen percent of eighth-grade students reported that they had used computers for at least 5 but less than 7 years. However, 9 percent of eighth-grade students had never used computers or had used them for less than one year, meaning they had only started using computers when they reached eighth grade.



Overall, responses of eighth-grade students showed that some had more years of experience using computers than others. Although there were differences in students’ use of ICT for school-related purposes, most students felt confident using ICT.

It should be noted that the data presented here were collected in 2018; any changes since then due to the coronavirus pandemic or other factors are not reflected in the results reported here. The NAEP TEL and ICILS samples both include public and private schools. Data reported in the text and figures are rounded to the nearest integer.

 

Resources for more information:

 

By Yan Wang, AIR, and Taslima Rahman, NCES

Facilitating Causal Research in CTE: Notes from the Network

The Career and Technical Education (CTE) Research Network just entered its third year, and it is time to share recent accomplishments with the IES community! As a reminder, the CTE Research Network (led by the American Institutes for Research, or AIR) was created to expand the evidence base on the impact of CTE programs on student outcomes using causal research methods.

 

Research

In June 2020, AIR released a preliminary report of CTE sites that are ready for causal evaluation. Designed to support researchers interested in studying the impact of CTE on student outcomes, the report details the history, theory of change, student enrollments, and other information for 4 selected CTE sites around the country. The Network hopes that researchers will use the information in the report to design evaluation studies of these programs.

In July 2020, a fifth research project joined the network. A team from MDRC, led by Rachel Rosen, was recently funded by IES to study the Impact of Technology-Based Career Advising Tools on High School Students' CTE Choices and Academic Performance. In partnership with Communities in Schools (CIS), the study will use a three-arm, school-level random assignment research design (RCT) to assess the effects of Navience and YouScience on students' self-expressed attitude and interest in career pathways, CTE course taking patterns, and engagement with and progress towards graduation. We welcome the team to the Network and look forward to learning whether and how these advising tools influence student thinking about career options, choice of relevant CTE coursework and work-based learning options, and decisions about CTE concentration in available pathways and programs of study.

A small group of researchers from different Network teams collaborated on and recently released a technical working paper on counterfactuals in CTE. It can be challenging to identify comparison groups for CTE students because it is an elective into which they self-select. The paper describes a variety of rigorous methods of comparing CTE students to valid counterparts and provides case studies that illustrate how to use these methods.

 

Training

The CTE Research Network is committed to increasing the number of researchers trained to study CTE using causal methods. It is notoriously challenging to isolate the effects of CTE from other influences on student outcomes. In August 2020, the Network hosted 18 researchers for a week-long virtual summer training institute on how to design a causal study to examine the impact of CTE. During the week, participants learned how to implement randomized-control trials (RCTs), regression discontinuity designs (RDDs), and comparative interrupted time series (CITS) in a CTE context. After learning about each method, participants worked in small groups to apply the method to real data and had access to the instructors to ask questions. The feedback about the training was overwhelmingly positive. The lecture portions of the training will be posted soon to the training page of the Network’s website. Another week-long training institute will be held in summer 2021 (hopefully, in person!)

The Network is currently developing a series of online modules for CTE practitioners and state agency staff to strengthen capacity to access, conduct, understand, and use CTE research. There will be a presentation to preview the modules at the Association of Career and Technical Education (ACTE) Best Practices and Innovations Conference on October 9, 2020 and a longer and more in-depth session at ACTE’s Career Tech VISION conference the first week of December. These practitioner training modules will be available for free on the website in late Fall 2020.

 

Leadership and Dissemination

The CTE Research Network is regularly updating its resources page with publications of interest to the CTE research field. The most recent is a report of findings from MDRC’s study of P-Tech high schools. The Network’s equity workgroup (a group of researchers from across the Network’s member projects) also published a popular blog this summer on applying an equity lens to CTE research. The Network also posts outside resources such as a REL self-study tool on career readiness and evaluation reports from other researchers.

 


For more about the CTE Research Network, you can sign up to receive the Network’s quarterly newsletter at the bottom of their website’s home page and follow them on Twitter (@CTEResNetwork) and LinkedIn.

If you are interested in learning more about the CTE Research Network, contact the Director, Kathy Hughes (khughes@air.org).

If you are interested in discussing CTE research opportunities at IES, contact Corinne Alfeld (Corinne.Alfeld@ed.gov).

What National and International Assessments Can Tell Us About Technology in Students’ Learning: Eighth-Graders’ Readiness to Use Technology

Across the country in 2020, students, teachers, and parents have had to adapt to changes in the delivery of education instruction due to the coronavirus pandemic and turn to information and communication technologies (ICT) to learn, interact, and assess progress. It is important that we are able to assess students’ abilities to understand and use technology now more than ever. This post, the first in a three-part series, discusses the results of two technology-focused assessments, NAEP TEL and ICILS, which NCES administered in 2018 (see textbox for more information about these assessments).

Students’ performance on NAEP TEL and ICILS

According to the 2018 NAEP TEL, 46 percent of U.S. eighth-grade students scored at or above the NAEP Proficient level, meaning that they were able to demonstrate the selection and use of an appropriate range of tools and media (figure 1). According to the 2018 ICILS, 25 percent of eighth-grade students scored at or above proficiency level 3 for computer and information literacy—that is, they demonstrated the capacity to work independently when using computers as information-gathering and management tools. In addition, 20 percent of eighth-grade students scored in the upper region for computational thinking, meaning that they demonstrated an understanding of computation as a problem-solving framework.


Figure 1. Percentage of eighth-grade students identified as at or above proficient, by assessment: 2018


There are many factors that may affect performance on these assessments, such as access to technology, devices, hardware, and software; access to learning opportunities using technology; amount of experience using technology; and attitudes toward technology.

Student’s participation in technology-related classes

For NAEP TEL, students were asked if they were currently taking a technology-related class or if they had taken one in the past. In 2018, about 57 percent of U.S. eighth-grade students were either currently enrolled in or had taken at least one technology-related class, such as industrial technology, engineering, or a class that involved learning to use, program, or build computers (table 1). In addition, a higher percentage of students who had completed a technology-related class before or during eighth-grade scored at or above the NAEP Proficient level, compared with students who had not completed such a class (table 2).




Overall, both assessments show that a portion of students demonstrated the knowledge and skills necessary to meet the assessment’s defined proficiency measures, although more than half reported taking technology-related classes before or during eighth grade.

It should be noted that the data presented here were collected in 2018; any changes since then due to the coronavirus pandemic or other factors are not reflected in the results reported here. The NAEP TEL and ICILS samples both include public and private schools, but the two assessments use different methods for reporting student performance. Data from these assessments do not support any causal inferences as they are not experimental studies. Data reported in the text and figures are rounded to the nearest integer.

 

Resources for more information:

By Mary Ann Fox, AIR, and Taslima Rahman, NCES

Listening to Schools: The National Teacher and Principal Survey (NTPS) Shares Educators’ Perspectives on Coronavirus Impacts on Education

As the 2020–21 school year gets underway, many are considering the tremendous impact the coronavirus will have on classrooms—whether in person or virtual—across the United States. What can be done to support policymakers and education sector leaders as they strive to address, amongst other concerns, potentially unequitable learning opportunities and mental health challenges?

The National Teacher and Principal Survey (NTPS) will gather critical information from teachers and principals about the changes implemented and lessons learned by schools and their staff during the 2020–21 school year. The NTPS is a nationwide system of related surveys that collect data on elementary and secondary education in the United States, including teaching and working conditions in schools and characteristics of public and private school teachers and principals at the state level.

Conducted every 2 to 3 years, NTPS provides critical representative data to policymakers and researchers on school organization, staff evaluations, teacher and principal preparation and professional development, classes taught, school characteristics, demographics of the teacher and principal labor force, and other important education topics. These data serve to inform those who set funding and other priorities, including Congress, the U.S. Department of Education, state education agencies, and public school districts. The data also allow important comparative analyses of key education personnel in public and private educational settings.

Unlike many other studies capturing information about the coronavirus and related education issues, NTPS allows for comparisons at the national level, between states for public schools and by affiliation for private schools, and over time. It is important that NTPS questionnaires reach selected teachers, principals, and other staff during these changing times. The information they provide will help decision makers evaluate the effects of school workplace conditions, salaries, and training opportunities on the educational workforce and aid in the U.S Department of Education’s program planning in the areas of teacher recruitment and retention, teaching policies, and teacher education.

But participation is key! If school staff do not participate in NTPS when selected, the data will provide an incomplete and possibly misleading description of the impact of the coronavirus on school communities, potentially affecting funding and other policy decisions.  

 

What do we know about the coronavirus and the 202021 school year?

By May 2020, in just the United States alone, at least 50.8 million public school students were affected by ordered or recommended K–12 school closures. According to Education Week reports, school districts unveiled a variety of reopening plans for the 2020–21 school year that include remote, hybrid or partial in-person, or full in-person learning approaches. As of September 2, 73 percent of the 100 largest school districts had announced they wiould resume with remote learning only. Information from the Census Bureau’s experimental weekly Household Pulse Survey suggests that students in one of every six households do not usually have access to the internet for education purposes.

Regardless of each school district’s decision for the beginning of the school year, local and state education leaders are responsible for numerous decisions on behalf of their schools, students, teachers, and school staff that rely heavily on the availability of reliable data. But most of the existing information on the education sector’s response to the coronavirus is at the district or state level and does not typically include information about experiences of teachers and principals directly from these critical education providers. Also, because of varying reporting resources and practices across the nation’s 130,000 public and private K–12 schools, a consistent national-level understanding of coronavirus-related education problems is not readily available. As a result, key policymakers and other decision makers currently have little information from individual teachers or principals about coronavirus-related problems specifically and education issues more generally. NTPS provides an opportunity for the voices of teachers, principals, and other school staff to be included in the conversation.

 

How can NTPS show what is happening in K–12 schools across the United States?

Last administered in 2017–18, the NTPS is set to resume in the 2020–21 school year and new questions have been added to reflect changes that may have occurred due to the coronavirus. This offers an opportunity for sampled teachers, principals, and schools to provide valuable data that explain their experience as educators during the coronavirus pandemic. In addition to new items, data gleaned from recurring questions will capture changes over time and yield important insights into areas of success and areas in need of further support. Data from prior school years has already been used during the pandemic to highlight differences in the number of health staff, such as school nurses, and mental health staff, such as counselors, psychologists, and social workers. Gathering more responses to these and other questions will allow for trend analyses, giving policymakers and other decision makers a better understanding of changes occurring at the teacher, principal, and school levels.

 

How will the survey be conducted?

The NTPS data collection process is both voluntary and self-administered, meaning all questionnaires can be completed without any in-person contact and without interruption for staff who may be fully working remotely. Teachers, principals, and schools who have been sampled to participate in the survey will be contacted by mail and e-mail and invited to complete the questionnaires online. Sampled participants will also receive paper surveys at their school mailing addresses.

 

Why is this survey important?

Responses from sampled schools ensure that NTPS estimates are reliable and accurately reflect the activities of all U.S. public and private schools. These data are vital as policymakers, researchers, families, and school staff strive to understand and respond to the effects of the current pandemic and build a better, stronger education sector for the future—including improved response options for potential future pandemics. NTPS responses during the 2020–21 school year can be compared to data from future NTPS cycles to understand possible longer-term impacts of the current significant changes to education delivery in the country and across the states. These data provide national and state policymakers with a distinct understanding of the condition of K–12 education in their communities and will remain important as leaders monitor changes in the education sector in future years.

For more information about the National Teacher and Principal Survey (NTPS), please visit https://nces.ed.gov/surveys/ntps/.

 

By Julia Merlin, NCES