Inside IES Research

Improving the social and emotional climate of schools has become a growing priority for educators and policymakers in the past decade. The prevailing strategies for improving school climate include social and emotional learning, positive behavioral supports, and trauma-informed approaches. Many of these strategies foreground the importance of students having a voice in intervention, as students are special experts in their own social and emotional milieus.

Parallel to this trend has been a push toward student-centered pedagogical approaches in high schools that are responsive to cultural backgrounds and that promote skills aligned with the demands of the modern workplace, like critical thinking, problem-solving, and collaboration. Culturally responsive and restorative teaching and problem- and project-based learning are prominent movements. In this guest blog, Dr. Adam Voight at Cleveland State University discusses an ongoing IES-funded Development and Innovation project taking place in Cleveland, Ohio that aims to develop and document the feasibility of a school-based youth participatory action research intervention.

Our project is exploring how youth participatory action research (YPAR) may help to realize two objectives—school climate improvement and culturally-restorative, engaged learning. YPAR involves young people leading a cycle of problem identification, data collection and analysis, and evidence-informed action. It has long been used in out-of-school and extracurricular spaces to promote youth development and effect social change. We are field testing its potential to fit within more formal school spaces.

Project HighKEY

The engine for our project, which we call Project HighKEY (High-school Knowledge and Education through YPAR), is a design team composed of high school teachers and students, district officials, and university researchers. It is built from the Cleveland Alliance for Education Research, a research-practice partnership between the Cleveland Metropolitan School District, Cleveland State University, and the American Institutes for Research. The design team meets monthly to discuss YPAR theory and fit with high school curriculum and standards and make plans for YPAR field tests in schools. We have created a crosswalk of the documented competencies that students derive from YPAR and high school standards in English language arts (ELA), mathematics, science, and social studies in Ohio. For example, one state ELA standard is “Write arguments to support claims in an analysis of substantive topics or texts, using valid reasoning and relevant and sufficient evidence,” and through YPAR students collect and analyze survey and interview data and use their findings to advocate for change related to their chosen topic. A state math standard is “Interpret the slope and the intercept of a linear model in the context of data,” and this process may be applied to survey data students collect through YPAR, making an otherwise abstract activity more meaningful to students.  

Assessing the Effectiveness of YPAR

Remaining open-minded about the various ways in which YPAR may or may not fit in different high school courses, we are currently testing its implementation in a pre-calculus course, a government course, an English course, and a life-skills course. For example, a math teacher on our design team has built her statistics unit around YPAR. Students in three separate sections of the course have worked in groups of two or three to identify an issue and create a survey that is being administered to the broader student body. These issues include the lack of extracurricular activities, poor school culture, and unhealthy breakfast and lunch options. Their survey data will be used as the basis for learning about representing data with plots, distributions, measures of center, frequencies, and correlation after the winter holiday. Our theory is that students will be more engaged when using their own data on topics of their choosing and toward the goal of making real change. Across all of our project schools, we are monitoring administrative data, student and teacher survey data, and interview data to assess the feasibility, usability, and student and school outcomes of YPAR.

Impact of COVID-19 and How We Adapted

We received notification of our grant award in March 2020, the same week that COVID-19 shut down K-12 schools across the nation. When our project formally began in July 2020, our partner schools were planning for a wholly remote school year, and we pivoted to hold design team meetings virtually and loosen expectations for teacher implementation. Despite these challenges, several successful YPAR projects during that first year—all of which were conducted entirely remotely—taught all of us much about how YPAR can happen in online spaces. This school year, students and staff are back to in-person learning, but, in addition to the ongoing pandemic, the crushing teacher shortage has forced us to continue to adapt. Whereas we once planned our design team meeting during the school day, we now meet after school due to a lack of substitute teachers, and we use creative technology to allow for mixed virtual and in-person attendance. Our leadership team is also spending a great deal of time in classrooms with teachers to assist those implementing for the first time. Our goal is to create a resource that teachers anywhere can use to incorporate YPAR into their courses. The product will be strengthened by the lessons we have learned from doing this work during these extraordinary times and the resulting considerations for how to deal with obstacles to implementation.

Adam Voight is the Director of the Center for Urban Education at Cleveland State University.

For questions about this grant, please contact Corinne Alfeld, NCER Program Officer, at Corinne.Alfeld@ed.gov.

Under an IES grant, Michigan State University, in collaboration with the Michigan Department of Education, the Michigan Center for Educational Performance and Information, and the University of Michigan, is assessing the implementation, impact, and cost of the Michigan “Read by Grade 3” law intended to increase early literacy outcomes for Michigan students. In this guest blog, Dr. Tanya Wright and Lori Bruner discuss how they were able to quickly pivot to a remote data collection plan when COVID-19 disrupted their initial research plan.  

The COVID-19 pandemic began while we were planning a study of early literacy coaching for the 2020-2021 academic year. It soon became abundantly clear that restrictions to in-person research would pose a major hurdle for our research team. We had planned to enter classrooms and record videos of literacy instruction in the fall. As such, we found ourselves faced with a difficult choice: we could pause our study until it became safer to visit classrooms and miss the opportunity to learn about literacy coaching and in-person classroom instruction during the pandemic, or we could quickly pivot to a remote data collection plan.

Our team chose the second option. We found that there are multiple technologies available to carry out remote data collection. We chose one of them (a device known as the Swivl) that included a robotic mount, where a tablet or smartphone can be placed to take the video, with a 360-degree rotating platform that works in tandem with a handheld or wearable tracker and an app that allows videos to be instantly uploaded to a cloud-based storage system for easy access.

Over the course of the school year, we captured over 100 hours of elementary literacy instruction in 26 classrooms throughout our state. While remote data collection looks and feels very different from visiting a classroom to record video, we learned that it offers many benefits to both researchers and educators alike. We also learned a few important lessons along the way.

First, we learned remote data collection provides greater flexibility for both researchers and educators. In our original study design, we planned to hire data collectors to visit classrooms, which restricted our recruitment of schools to a reasonable driving distance from Michigan State University (MSU). However, recording devices allow us to capture video anywhere, including rural areas of our state that are often excluded from classroom research due to their remote location. Furthermore, we found that the cost of purchasing and shipping equipment to schools is significantly less than paying for travel and people’s time to visit classrooms. In addition, using devices in place of data collectors allowed us to easily adapt to last-minute schedule changes and offer teachers the option to record video over multiple days to accommodate shifts in instruction due to COVID-19.

Second, we discovered that we could capture more classroom talk than when using a typical video camera. After some trial and error, we settled on a device with three external wireless microphones: one for the teacher and two additional microphones to place around the classroom. Not only did the extra microphones record audio beyond what the teacher was saying, but we learned that we can also isolate each microphone during data analysis to hear what is happening in specific areas of the classroom (even when the teacher and children were wearing masks). We also purchased an additional wide-angle lens, which clipped over the camera on our tablet and allowed us to capture a wider video angle.  

Third, we found remote data collection to be less intrusive than sending a research team into schools. The device is compact and can be placed on any flat surface in the classroom or be mounted on a basic tripod. The teacher has the option to wear the microphone on a lanyard to serve as a hands-free tracker that signals the device to rotate to follow the teacher’s movements automatically. At the end of the lesson, the video uploads to a password-protected storage cloud with one touch of a button, making it easy for teachers to share videos with our research team. We then download the videos to the MSU server and delete them from our cloud account. This set-up allowed us to collect data with minimal disruption, especially when compared to sending a person with a video camera to spend time in the classroom.

As with most remote work this year, we ran into a few unexpected hurdles during our first round of data collection. After gathering feedback from teachers and members of our research team, we were able to make adjustments that led to a better experience during the second round of data collection this spring. We hope the following suggestions might help others who are considering such a device to collect classroom data in the future:

1. Consider providing teachers with a brief informational video or offering after-school training sessions to help answer questions and address concerns ahead of your data collection period. We initially provided teachers with a detailed user guide, but we found that the extra support was key to ensuring teachers had a positive experience with the device. You might also consider appointing a member of your research team to serve as a contact person to answer questions about the remote data collection during data collection periods.
2. As a research team, it is important to remember that team members will not be collecting the data, so it is critical to provide teachers with clear directions ahead of time: what exactly do you want them to record? Our team found it helpful to send teachers a brief two-minute video outlining our goals and then follow up with a printable checklist they could use on the day they recorded instruction. 
3. Finally, we found it beneficial to scan the videos for content at the end of each day. By doing so, we were able to spot a few problems, such as missing audio or a device that stopped rotating during a lesson. While these instances were rare, it was helpful to catch them right away, while teachers still had the device in their schools so that they could record missing parts the next day.

Although restrictions to in-person research are beginning to lift, we plan to continue using remote data collection for the remaining three years of our project. Conducting classroom research during the COVID-19 pandemic has proven challenging at every turn, but as we adapted to remote video data collection, we were pleased to find unanticipated benefits for our research team and for our study participants.

This blog is part of a series focusing on conducting education research during COVID-19. For other blog posts related to this topic, please see here.

Tanya S. Wright is an Associate Professor of Language and Literacy in the Department of Teacher Education at Michigan State University.

Lori Bruner is a doctoral candidate in the Curriculum, Instruction, and Teacher Education program at Michigan State University.

The IES-funded R&D Center, the Center for the Success of English Learners (C-SEL), is undertaking a focused program of research aimed at improving access and outcomes for English Learners. One of C-SEL’s recent activities has been to develop resources to aid policymakers and practitioners working with middle school and secondary English learners. The research team at the Center, including Drs Diane August and Coleen Carlson, along with Maria Yolanda Cieslak and Kenneth Michael Nieser, recently released a brief on Educating English Learner Students During the Pandemic: Remote & In-person Instruction & Assessment-Recommendations and Resources for State and Districts. In this guest blog, the researchers provide an overview of the brief.

English Learners (ELs) benefit from specialized support to help them acquire second language proficiency and core content knowledge that builds on their cultural and linguistic assets. This specialized support is required by law, and the U.S. Department of Education reminded States that this is the case, even when learning is remote. Based on a review of the existing relevant literature, this brief provides detailed information related to the impact of remote learning on English Learners (ELs) and their teachers during the pandemic and the potential and limitations of using digital learning resources (DLRs) to educate these students. For instance, while DLRs have the potential to support learning and engagement for ELs, districts report barriers to their use, such as lack of home access to DLRs; teachers’ level of expertise and technology skills; and the lack of knowledge around what are the appropriate DLRs for ELs.

The brief also describes current legislation that authorizes funds for a variety of activities that could be used to support ELs and their families when instruction is delivered remotely. Some of these federal resources include:

• Formula Grant Programs Under the Every Student Succeeds Act
• CARES Act
• Coronavirus Response and Relief Supplemental Appropriations Act

In addition, the brief includes information about policies and resources for each of the 50 states in place in December 2020 that support districts and schools in instructing and assessing EL students remotely. It also includes a reading list of recent resources focused on remote learning for ELs, with brief descriptions of each resource and links to the resource.

In March 2021, the Center hosted a webinar to discuss recent recommendations for states and districts. One set of recommendations focuses on methods to enhance the learning and emotional well-being of EL students who have lost ground during the pandemic. Recommendations are also made for assessing ELs when schooling is or has been remote or hybrid. These recommendations can be found in the brief.  

In the upcoming months, C-SEL investigators look forward to preparing future blog posts and research briefs on the research of the Center and the students and teachers we are serving. Next up on the agenda is an overview of the students, highlighting their diversity, and some too often ignored, forgotten, or simply unknown characteristics of this important subgroup. Stay tuned!

Dr. Diane August is Principal at D. August and Associates and a Senior Research Scientist at the Center for Applied Linguistics.

Dr. Coleen Carlson is an Associate Research Professor, and Associate Director at the Texas Institute for Measurement, Evaluation, and Statistics (TIMES) at the University of Houston.

Ms. Maria Yolanda Cieslak is a Professional Development Specialist at the Center for Applied Linguistics.

Mr. Kenneth Michael Nieser is a Researcher at the Texas Institute for Measurement, Evaluation, and Statistics at the University of Houston.

Under an IES grant, Drs. Leslie Babinski, Steve Amendum, Steve Knotek, and Marta Sánchez are evaluating the impact of the Developing Consultation and Collaboration Skills (DCCS) program. The DCCS program is a year-long professional development intervention designed to support English-as-a-Second Language (ESL) and classroom teachers to develop their skills in collaboration, literacy instruction, and parent outreach and engagement for their Latino students. Unfortunately, COVID-19 disrupted their study, but rather than calling it quits, the research team used it as an opportunity to explore how teachers are responding to a new normal and learned some important lessons along the way.

Last spring, we were in the middle of conducting a randomized control trial in elementary schools working with teaching teams of kindergarten, first grade, and ESL teachers when COVID-19 hit, and schools pivoted to online instruction.

As we paused our efficacy trial and adapted our plans, we continued to stay engaged with teachers despite the rapidly shifting circumstances. We realized that the foundational skills central to our intervention—using high-impact instructional strategies for English learners (ELs), building on families’ cultural wealth, promoting collaboration between ESL and classroom teachers, and ongoing, supportive implementation coaching—could be modified and adapted for different contexts and modes of delivery. These ongoing interactions with teachers provided us with an up-close look at their experiences as they connected with families remotely and adapted to this new and unfamiliar way of teaching. We learned about the challenges, successes, and possibilities for providing online instruction for young ELs.

While online teaching during the current school year looks and feels very different from the emergency shift to remote teaching from last spring, we learned important lessons from the experiences of teachers and students.

First, we learned that our implementation coaching model could be productive in a virtual format. In fact, teachers were eager to engage in reflective conversations despite the demands on their time and the stress of the pandemic. Each of the school teams continued to meet with our implementation coach. Together, they found new ways to scaffold instruction for ELs during online learning. Teachers supported one another in finding creative ways to use technology to communicate with families and provide support that reached beyond academics. They also created lessons that intentionally adapted instruction to reach ELs in their classrooms. For example, embedded in the videos the teachers created, they continued to use our core instructional strategies, such as previewing academic vocabulary before reading a new text, supplementing student background knowledge related to the content, providing sentence frames for both oral and written participation, and selecting texts related to student cultures and families.

Second, it is clear that there are serious inequities in access to online schooling. In our study, even as teachers made the commendable efforts described above, they also reported that more than half of their EL students required a device from the school district, while one-third of the families also needed a Wi-Fi hotspot to access online learning. This rate is considerably higher than a Pew Research Center poll that found that one in five parents reported difficulties with online learning due to lack of a computer or Wi-Fi access, highlighting the fact that many English Learners may have limited access to the technology necessary for online learning.

Third, online learning requires a partnership between teachers, families, and other caregivers. Even with a device and internet access, many young children and their families had difficulty logging onto their school accounts and navigating the technical aspects of the learning platforms for online schooling. Teachers reported that about half of the EL students in their classrooms had a parent or guardian who was able to help them with schoolwork at home. About 20% had help from another adult or a sibling. In one case, a teacher described a family in which a kindergarten student received support for online schooling from her brother in second grade. In our study, support at home was critical for student participation in online lessons. Our work with teachers during remote teaching and learning highlighted how parents and other caregivers are important advocates for their children’s education and are eager to partner with teachers and schools to help their children succeed.

Looking to the future, it is clear that instruction for ELs will need to focus on equitable access to high-quality instruction, whether online or in-person. Access to devices and Wi-Fi is essential, not only for online teaching and learning, but also for extending learning into the home. Finally, we note that teacher collaboration and coaching can be effectively adapted for an online environment and is an essential component in providing support to teachers for high-quality instruction for ELs.

Although we could not have anticipated the value of continuing to work with teachers during the pivot to remote instruction, we are grateful for the experience and all that we have learned during the process.

For more information, see this EdWeek article and this interview on The TakeAway.

Dr. Leslie Babinski is an associate research professor in the Sanford School of Public Policy and the Director of the Center for Child and Family Policy.

Dr. Steve Amendum is a professor at the School of Education at the University of Delaware.

Dr. Steve Knotek is a professor in the School of Education at the University of North Carolina at Chapel Hill.

Dr. Marta Sánchez is an associate professor in the Watson College of Education at the University of North Carolina Wilmington.

This is part of a series of blog posts focusing on conducting education research during COVID-19.

Under an IES grant, the RAND Corporation, in collaboration with NWEA, is developing strategies for schools and districts to address the impacts of COVID-19 disruptions on student assessment programs. The goal is to provide empirical evidence of the strengths and limitations of strategies for making decisions in the absence of assessment data. Jonathan Schweig, Andrew McEachin, and Megan Kuhfeld describe early findings from surveys and structured interviews regarding key concerns of districts and schools. 

As a first step, we surveyed assessment and research coordinators from 23 school districts (from a sample of 100 districts) and completed follow-up interviews with seven of them on a variety of topics, including the re-entry scenario for their district, the planning activities that they were not able to perform this year due to coronavirus-based disruptions to spring 2020 assessments, and the strategies they were employing to support instructional planning in the absence of assessment data. While the research is preliminary and the sample of respondents is not nationally representative, the survey and interview responses identified two key concerns arising from the lack of spring 2020 assessment data which has made it challenging to examine student or school status and change over time, especially as COVID-19 has differential impacts on student subgroups:

• Making course placement decisions. Administrators typically rely on spring assessment scores—often in conjunction with other assessment information, course grades, and teacher recommendations—to make determinations for course placements, such as who should enroll in accelerated or advanced mathematics classes. 
• Evaluating programs or district-wide initiatives. Many districts monitor the success of these programs internally by looking at year-to-year change or growth for schools or subgroups of interest. 

How are school systems responding to these challenges? Not surprisingly, the responses vary depending on local contexts and resources. Where online assessments were not feasible in spring 2020, some school districts used older testing data to make course recommendations, either from the winter or from the previous school year. Some districts relaxed typical practice and provided more autonomy to individual schools, relying on school staff to exercise local judgment around course placements and using metrics like grades and teacher recommendations. Other districts reported projecting student scores based on student assessment histories. Relatedly, some districts were already prepared for this decision because they had recently experienced difficulties with adopting an online assessment system and had to address similar problems caused by large numbers of missing or invalid tests.

School districts also raised concerns about whether assessments administered during the 2020-21 school year would be valid and comparable so that they could be used in student placement and program evaluation decisions. These concerns included the following:

• Several respondents raised concerns about the trustworthiness of remote assessment data collected this fall and the extent to which results could be interpreted as valid indicators of student achievement or understanding.
• Particularly for districts that started the 2020-21 school year remotely, respondents were concerned about student engagement and motivation and the possibility of students rushing assessments, running into technological or internet barriers, or seeking assistance from guardians or other resources. 
• Respondents raised questions about the extent to which available assessment scores are representative of school or district performance as a whole. Given that vulnerable students (for example, students with disabilities, students experiencing homelessness) may be the least likely to have access to remote instruction and assessments, it is likely that the students who are not assessed this year are different from students who are able to be assessed.
• Other respondents noted that they encountered resistance from parents around fall assessment because they prioritized student well-being (for example, safety, sense of community, and social and emotional well-being) more so than academics. This is a perspective that resonates with recent findings from a nationally representative sample of teachers and school leaders drawn from RAND’s American Educator Panel (AEP).

In the next phase of the work, the research team plans to:

• Conduct a series of simulation and empirical studies regarding the most common strategies that the district respondents indicated they were using to make course placement decisions and to evaluate programs or district-wide initiatives.
• Provide a framework to help guide local research on the intended (and unintended) consequences for school and school system decision making when standardized test scores are not available.

We welcome individuals to reach out to RAND with additional recommendations or considerations. We are also interested in hearing how districts are approaching course placement, accountability, and program evaluation across the country. Connect with the research team via email at jschweig@rand.org.

Jonathan Schweig is a social scientist at the nonprofit, nonpartisan RAND Corporation.

Andrew McEachin is a senior policy researcher at the nonprofit, nonpartisan RAND Corporation.

Megan Kuhfeld is a researcher at NWEA.