Inside IES Research

Notes from NCER & NCSER

How Remote Data Collection Enhanced One Grantee’s Classroom Research During COVID-19

Under an IES grant, Michigan State University, in collaboration with the Michigan Department of Education, the Michigan Center for Educational Performance and Information, and the University of Michigan, is assessing the implementation, impact, and cost of the Michigan “Read by Grade 3” law intended to increase early literacy outcomes for Michigan students. In this guest blog, Dr. Tanya Wright and Lori Bruner discuss how they were able to quickly pivot to a remote data collection plan when COVID-19 disrupted their initial research plan.  

The COVID-19 pandemic began while we were planning a study of early literacy coaching for the 2020-2021 academic year. It soon became abundantly clear that restrictions to in-person research would pose a major hurdle for our research team. We had planned to enter classrooms and record videos of literacy instruction in the fall. As such, we found ourselves faced with a difficult choice: we could pause our study until it became safer to visit classrooms and miss the opportunity to learn about literacy coaching and in-person classroom instruction during the pandemic, or we could quickly pivot to a remote data collection plan.

Our team chose the second option. We found that there are multiple technologies available to carry out remote data collection. We chose one of them (a device known as the Swivl) that included a robotic mount, where a tablet or smartphone can be placed to take the video, with a 360-degree rotating platform that works in tandem with a handheld or wearable tracker and an app that allows videos to be instantly uploaded to a cloud-based storage system for easy access.

Over the course of the school year, we captured over 100 hours of elementary literacy instruction in 26 classrooms throughout our state. While remote data collection looks and feels very different from visiting a classroom to record video, we learned that it offers many benefits to both researchers and educators alike. We also learned a few important lessons along the way.

First, we learned remote data collection provides greater flexibility for both researchers and educators. In our original study design, we planned to hire data collectors to visit classrooms, which restricted our recruitment of schools to a reasonable driving distance from Michigan State University (MSU). However, recording devices allow us to capture video anywhere, including rural areas of our state that are often excluded from classroom research due to their remote location. Furthermore, we found that the cost of purchasing and shipping equipment to schools is significantly less than paying for travel and people’s time to visit classrooms. In addition, using devices in place of data collectors allowed us to easily adapt to last-minute schedule changes and offer teachers the option to record video over multiple days to accommodate shifts in instruction due to COVID-19.

Second, we discovered that we could capture more classroom talk than when using a typical video camera. After some trial and error, we settled on a device with three external wireless microphones: one for the teacher and two additional microphones to place around the classroom. Not only did the extra microphones record audio beyond what the teacher was saying, but we learned that we can also isolate each microphone during data analysis to hear what is happening in specific areas of the classroom (even when the teacher and children were wearing masks). We also purchased an additional wide-angle lens, which clipped over the camera on our tablet and allowed us to capture a wider video angle.  

Third, we found remote data collection to be less intrusive than sending a research team into schools. The device is compact and can be placed on any flat surface in the classroom or be mounted on a basic tripod. The teacher has the option to wear the microphone on a lanyard to serve as a hands-free tracker that signals the device to rotate to follow the teacher’s movements automatically. At the end of the lesson, the video uploads to a password-protected storage cloud with one touch of a button, making it easy for teachers to share videos with our research team. We then download the videos to the MSU server and delete them from our cloud account. This set-up allowed us to collect data with minimal disruption, especially when compared to sending a person with a video camera to spend time in the classroom.

As with most remote work this year, we ran into a few unexpected hurdles during our first round of data collection. After gathering feedback from teachers and members of our research team, we were able to make adjustments that led to a better experience during the second round of data collection this spring. We hope the following suggestions might help others who are considering such a device to collect classroom data in the future:

  1. Consider providing teachers with a brief informational video or offering after-school training sessions to help answer questions and address concerns ahead of your data collection period. We initially provided teachers with a detailed user guide, but we found that the extra support was key to ensuring teachers had a positive experience with the device. You might also consider appointing a member of your research team to serve as a contact person to answer questions about the remote data collection during data collection periods.
  2. As a research team, it is important to remember that team members will not be collecting the data, so it is critical to provide teachers with clear directions ahead of time: what exactly do you want them to record? Our team found it helpful to send teachers a brief two-minute video outlining our goals and then follow up with a printable checklist they could use on the day they recorded instruction. 
  3. Finally, we found it beneficial to scan the videos for content at the end of each day. By doing so, we were able to spot a few problems, such as missing audio or a device that stopped rotating during a lesson. While these instances were rare, it was helpful to catch them right away, while teachers still had the device in their schools so that they could record missing parts the next day.

Although restrictions to in-person research are beginning to lift, we plan to continue using remote data collection for the remaining three years of our project. Conducting classroom research during the COVID-19 pandemic has proven challenging at every turn, but as we adapted to remote video data collection, we were pleased to find unanticipated benefits for our research team and for our study participants.


This blog is part of a series focusing on conducting education research during COVID-19. For other blog posts related to this topic, please see here.

Tanya S. Wright is an Associate Professor of Language and Literacy in the Department of Teacher Education at Michigan State University.

Lori Bruner is a doctoral candidate in the Curriculum, Instruction, and Teacher Education program at Michigan State University.

IES Announces a New Research and Development Center for Self-Directed Learning Skills in Online College Courses

In response to a call from IES for research on how to best support postsecondary teachers and students to thrive in online environment, NCER is establishing a new research and development (R&D) center. This center, led by SRI International (SRI) and the Community College Research Center (CCRC) at Teachers College (Columbia University), aims to help faculty embed support for self-directed learning skills into their online and hybrid courses.

This R&D center will support postsecondary instructors in making optimal use of technology features often available in online course tools to bolster student self-management strategies. Through its research and capacity-building programs, the center aims to strengthen teaching and learning, improve student outcomes, and ensure all students—regardless of race, ethnicity, or socioeconomic status—have equitable learning opportunities and attainment in broad-access institutions.

“Lack of self-directed learning skills can hinder a student’s success in any college course,” says SRI’s Rebecca Griffiths, a lead researcher in the new center, “but the challenge is greater in online courses, which typically place more responsibility on students to manage their own learning.”

 

Self-directed learning skills, also known as self-regulated learning skills, encompass three interrelated and mutually reinforcing areas:

  • Affect, which includes self-efficacy and the motivation to learn
  • Strategic actions, which include planning, goal setting, strategies to organize, code, and rehearse
  • Metacognition, which includes self-monitoring, self-evaluating and self-correction

 

These three areas can form a virtuous cycle. When students believe that studying helps them learn important and useful knowledge, they are more likely to study strategically and effectively. Effective study habits in turn enhance academic performance and build positive mindsets including confidence, motivation around learning, and a sense of personal growth.

SRI and CCRC will partner with Achieving the Dream and nine broad-access, public colleges, and universities across the U.S. to conduct these research program activities.

 

The research goals of the R&D center are to—

  • Generate new knowledge about how faculty can effectively use technology features and instructional practices in online STEM courses to create a positive feedback loop for students
  • Shed light on institutional policies and practices and instructional environments needed to support a coherent, intentional, and sustainable approach to helping students build self-directed learning skills across their coursework
  • Develop and pilot a technology-enabled, skills development model that will use technology features already widely available in learning management systems, adaptive courseware, and mobile apps to deliver instruction on these skills
  • Using research findings to inform the development of a rich, interactive toolkit to support institutions and faculty as they implement self-directed learning skills instruction at scale in online programs.

 

In addition to carrying out the research activities, the center will provide national leadership and capacity-building activities for postsecondary teaching and learning. Through partnership with Achieving the Dream, technology developers, researchers, education equity advocates, and others, the center will establish the critical importance of integrating self-directed learning into instruction to improve teaching and learning and improve equity in postsecondary student outcomes. They will also engage faculty, instructional designers, and educational technology developers to share knowledge and to co-develop and disseminate capacity-building resources that support teaching these skills and strategies. 


The center is led by Dr. Deborah Jonas (PI, SRI International, top photo), Dr. Nicole Edgecombe (Co-PI, Teachers College, Columbia University), Dr. Rebecca Griffiths (Co-PI, SRI International), and Dr. Neil Seftor (Co-PI, SRI International).

This blog was written by the R&D center team. For further information about the grant, contact the program officer: Dr. Meredith Larson.

IES Supported Intervention “INSIGHTS Into Children’s Temperament” is Featured at the 2021 ED Games Expo

The ED Games Expo is an annual showcase of game-changing innovations in education technology developed through programs at ED and across the federal government. Since 2013, the Expo has been an in-person event at venues across Washington, D.C. Because of COVID-19, the 2021 Expo will be an entirely virtual experience from June 1 to 5.

This year, the Expo will showcase more than 160 learning games and technologies and feature 35 different virtual EdTech events of interest to a broad audience of viewers. See the Agenda for the lineup for the Ed Games Expo.

 

ED Games Expo: Featuring INSIGHTS into Children’s Temperament

INSIGHTS into Children’s Temperament, an IES-supported intervention, is being featured at the Expo this year. INSIGHTS supports children’s social-emotional development and academic learning by helping teachers and parents see how differences in children’s behavior might reflect temperament/personality. Children work with the INSIGHTS puppets and learn that other children and adults react differently to the same situation due to their temperaments. IES has supported two randomized controlled trials (RCTs, the “gold standard” for claims of impact) of INSIGHTS – one in New York City and the other (ongoing) in rural Nebraska. Evidence from the NYC RCT and a longitudinal follow up indicate that children who participate in the INSIGHTS program during early elementary school experience better academic and social behavioral outcomes immediately following participation in the program, and these positive impacts persist into middle school. 

 

During the 2020 ED Games Expo, Sandee McClowry and her team performed an INSIGHTS lesson at the Kennedy Center to hundreds of attendees, including children, students, and families. INSIGHTS will be featured in this year’s ED Games Expo in three ways.

  • Tuesday, June 1 at 8PM Eastern: There will be an “ED Games Expo Kick Off Show” hosted by the puppets from the INSIGHTS intervention and the characters from the Between the Lions children’s television program. All of the characters will share information about the ED Games Expo while having a lot of fun and hijinks on a road trip to Washington, DC.  The Show will be introduced by the Secretary of Education, Miguel Cardona, and will also feature cameo appearances by IES, ED, and government team members.
  • Wednesday, June 2 from 9PM to 9:45PM Eastern: Sandee McClowry will be hosting a Master Class for Educators. The event will introduce all of the INSIGHTS friends, including Coretta the Cautious, Gregory the Grumpy, Fredrico the Friendly, and Hilary the Hard Worker. The video will provide practical guidance to educators on how to deliver the intervention in a classroom. The event will conclude with a rich and engaging discussion with expert practitioners about how INSIGHTS addresses the social and emotional learning of children, educators, and parents. Click Here to access the YouTube broadcast of the Master Class and set a reminder to watch on June 1.
  • Materials from INSIGHTS, including puppets that can be printed out and professional development resources for educators, will be available to try out during the Expo and in the month of June.

 

For URL links to watch the ED Games Expo Kick Off Show and Master Class for Educators, See the Agenda. For more information and on how to access the resources INSIGHTS intervention, see the website.


Written by Emily Doolittle (Emily.Doolittle@ed.gov), NCER Team Lead for Social Behavioral Research at IES

 

Gender Stereotypes in STEM: Emergence and Prevention

In 2018, Dr. Allison Master and co-PI Andrew Meltzoff were awarded a grant, Gender Stereotypes in STEM: Exploring Developmental Patterns for Prevention. This 4-year project explores how and when gender stereotypes about STEM career pathways emerge. The study also seeks to identify ways to mitigate the effects of such stereotypes, such as whether a growth mindset can lead to changes in student attitudes and outcomes toward STEM. As an undergraduate student majoring in microbiology at UCLA, Yuri Lin, virtual intern at NCER, was interested in learning more about gender inequalities and stereotypes in STEM education. She recently had a chance to talk with Dr. Master about her research and its implications for increasing STEM participation among women.

 

How is American culture affecting the STEM gender gap, and how does the US compare to other countries on this issue?

When children grow up in American culture, they see lots of TV shows and books where mathematicians, scientists, and engineers are men. STEM-based toys are also heavily marketed toward boys rather than girls. Some countries have begun changing the portrayal of gender stereotypes in the media. For example, the UK’s Advertising Standards Authority has recently started banning TV commercials that reinforce gender stereotypes. Some cross-national studies have shown that gender-STEM stereotypes favoring men are linked to women’s lower success and participation in STEM. The United States is one of many Western countries in which women have more equality and freedom to choose their careers but are much less likely to choose STEM careers than men. We still have a lot of work to do in the United States to break down barriers for women in STEM, and we need to focus on helping girls and women see the value in choosing pathways into STEM.

 

Why do you think it is important to examine growth mindset as a potential way to reduce the effects of stereotypes and increase STEM interest in students?

Growth mindsets are beliefs that personal characteristics can be changed, through effort or the right strategies. This is contrasted with fixed mindsets, which are beliefs that those characteristics can’t be changed. Growth mindsets are particularly helpful for struggling students. Students who have a growth mindset remain focused on learning rather than looking smart, believe effort is important, and stay resilient even when they experience setbacks. These attitudes translate into putting forth more effort and determination, which lead to greater success. In our project, we want to know if a growth mindset can help girls stay motivated in computer science, a subject that can have a steep learning curve. Girls in particular often get discouraged when they feel that they don’t have what it takes to succeed in STEM. We hope that teaching girls to have a growth mindset will protect them from these negative stereotypes and increase their confidence in themselves and their sense of belonging in computer science.

 

Considering that your project includes students from grades 1 to 12, how do you plan to share your findings with teachers, students, and policymakers? Are there differences in how you might communicate the information for different age groups?

As a developmental psychologist, I think it’s important to communicate the information about different age groups to everyone! It can be very valuable to frame student motivation in the broader context of how students are growing and changing. Students start to endorse stereotypes about computer science and engineering very early—Grades 1-3—so elementary school is a great time to start counteracting stereotypes by showing a broad representation of who enjoys and succeeds in STEM. We start to see big gender gaps in computer science interest during middle school, so this is a great time to have girls participate in fun and engaging coding classes. And we’ve already noted how important it is for girls in high school to have a growth mindset in their STEM classes.

We have different goals for communicating with teachers, parents, and policymakers. We know that teachers are very busy, so we try to condense things into the most important practical tips. We’ve made short videos and infographics about our research for teachers. For policymakers, we write policy briefs, which combines our research with other findings that are relevant to education policy. And when we talk to parents, we try to focus on the importance of the experiences they provide for their kids. We really value spreading the word about our research to make sure it reaches people who can use it to make a difference. For more information and access to the various resources, please visit the I AM Lab website.

 


Allison Master, PhD (@AllisonMaster), a developmental psychologist and an assistant professor at the University of Houston, has conducted extensive research on the development of motivation and identity in STEM education. 

Written by Yuri Lin (ylin010101@g.ucla.edu), intern for the Institute of Education Sciences and a Microbiology, Immunology, and Molecular Genetics major at UCLA.

Towards a Better Understanding of Middle-Schoolers’ Argumentation Skills

What is the difference between fact and opinion? How do you find relevant evidence and use it to support a position? Every day, teachers help students practice these skills by fostering critical discussions, a form of argumentation that encourages students to use reasoning to resolve differences of opinion.

In their IES-funded study, Exploring and Assessing the Development of Students' Argumentation Skills, Yi Song and her colleagues are uncovering activities (both teacher led and technology supported) that can improve middle-school students’ ability to generate better oral and written arguments.

This project began in 2019 and is working in classrooms and with teachers and students. The researchers have created a series of videos that describe their work. In this series, Dr. Song and her co-PIs, Dr. Ralph Ferretti and Dr. John Sabatini, discuss why the project is important to education, how they will conduct the research plan, and how educators can apply what they are learning in classrooms.

 

 


For questions and more information, contact Meredith Larson (Meredith.Larson@ed.gov), Program Officer, NCER