Inside IES Research

Notes from NCER & NCSER

Assessing Math Understanding of Students with Disabilities During a Pandemic

For almost two decades, IES/NCSER has funded Brian Bottge and his teams at the University of Kentucky and University of Wisconsin-Madison to develop and test the efficacy of a teaching method called Enhanced Anchored Instruction (EAI), which helps low-achieving middle school students with math disabilities develop their problem-solving skills by solving meaningful problems related to a real-world problem. The research findings support the efficacy of EAI, especially for students with math disabilities. Most recently, Bottge and his team have been researching innovative forms of assessment that more adequately capture what students with disabilities know both conceptually and procedurally in solving math problems. With supplemental funding, IES/NCSER extended Dr. Bottge’s latest grant to test the use of oral assessment to measure student knowledge and compare that with the knowledge demonstrated on a pencil and paper test. The COVID-19 pandemic introduced added challenges to this work when schools closed and students shifted to online education.

Below we share a recent conversation with Dr. Bottge about the experience of conducting research during a pandemic and what he and his team were still able to learn about the value of oral assessment in mathematics for students with disabilities.

What changes did you observe in the intervention implementation by teachers due to the COVID-related shift to online learning?

Photo of Dr. Brian Bottge

The shift to online learning created changes in class size and structure. For 38 days (22 days in classroom, 16 days online through a virtual meeting platform), the middle school special education teacher first taught concepts through a widely used video-based anchored problem, the Kim’s Komet episode of the Jasper Project, in which characters compete in a “Grand Pentathlon.” The teacher then engaged the students in a hands-on application of the concepts by running a live Grand Pentathlon. In the Grand Pentathlon, students make their own cars, race them on a full-size ramp, time them at various release points on the ramp, and graph the information to estimate the speed of the cars. The purpose of both units was to help students develop their informal understanding of pre-algebraic concepts such as linear function, line of best fit, variables, rate of change (slope), reliability, and measurement error. Midway through the study, in-person instruction was suspended and moved online. Instead of working with groups of three to four students in the resource room throughout the day, the teacher provided online instruction to 14 students at one time and scheduled one-on-one sessions with students who needed extra help.

What challenges did you observe in the students interacting with the activities and their learning once they shifted to online learning?

All students had access to a computer at home and they were able to use the online platform without much confusion because they had used it in other classes. The screen share feature enabled students to interact with much of the curriculum by viewing the activities, listening to the teacher, and responding to questions, although they could not fully participate in the hands-on part of the lessons. Class attendance and student behavior were unexpectedly positive during the days when students were online. For example, one student had displayed frequent behavioral outbursts in school but became a positive and contributing member of the online class. The ability to mute mics in the platform gave the teacher the option of allowing only one student to talk at a time.

Were students still able to participate in the hands-on activities that are part of the intervention?

For the hands-on activities related to the Grand Pentathlon competition, the teacher taught online and a research staff member manipulated the cars, track, and electronic timers from campus. Students watched their computer screens waiting for their turn to time their cars over the length of the straightaway. The staff member handled each student’s cars and one by one released them from the height on the ramp as indicated by each student. After students had recorded the times, the teacher asked students to calculate and share the speeds of their cars for each time trial height.

Do you have any other observations about the impact of COVID-19 on your intervention implementation?

One of the most interesting observations was parent participation in the lessons. Several parents went beyond simply monitoring how their child was doing during the units to actively working out the problems. Some were surprised by the difficulty level of the math problems. One mother jokingly remarked: I thought the math they were going to do was as easy as 5 + 5 = 10. The next time my son might have to be the parent and I might have to be the student. You all make the kids think and I like that.

When COVID-19 shut down your participating schools, how were you able to adjust your data collection to continue with your research?

We used the same problem-solving test that we have administered in several previous studies (Figure 1 shows two of the items). On Day 1 of the study (pre-COVID), students took the math pretest in their resource rooms with pencil and paper. Due to COVID-19 school closures, we mailed the posttest and test administration instructions to student homes. On the scheduled testing day during an online class session, students removed the test from the envelope and followed directions for answering the test questions while we observed remotely. On Days 2 and 3 of the study (pre-COVID), an oral examiner (OE) pretested individual students in person. The OE asked the student questions, prompting the student to describe the overall problem, identify the information needed for solving the problem, indicate how the information related to their problem-solving plan, and provide an answer. Due to COVID-19, students took the oral posttests online. The teacher set up a breakout room in the platform where the OE conducted the oral assessments and a second member of the research team took notes.

A picture depicting two sample questions. The first shows a graph of two running paths along with the text, "3. The total distance covered by two runners is shown in the graph below. a. How much time did it take runner 1 to go 1 mile? b. About how much time after the start of the race did one runner pass the other?" The second image features a marble on top of a ramp accompanied with the question "What is the speed of a marble (feet per second) let go from the top of the ramp? (Round your answer to the nearest tenth.)"Figure 1. Sample Items from the Problem-Solving Test

During the testing sessions, the OE projected each item on the students’ computer screens. Then she asked the student to read the problem aloud and describe how to solve it. The OE used the same problem-solving prompts as was used on the pretests. For problems that involved graphs or charts, the OE used the editing tools to make notations on the screen as the students directed. One challenge is that oral testing online made it more difficult to monitor behavior and keep students on task. For example, sometimes students became distracted and talked to other people in their house.

What were the results of this study of oral assessment in mathematics for students with disabilities?

Our results suggest that allowing students to describe their understanding of problems in multiple ways yielded depth and detail to their answers. We learned from the oral assessment that most students knew how to transfer the data from the table to an approximate location on the graph; however, there was a lack of precision due to a weak understanding of decimals. For item 4 in Figure 1, the use of decimals confused students who did not have much exposure to decimals prior to or during the study. We also found that graphics that were meant to help students understand the text-based items were in some cases misleading. The representation in item 4 was different than the actual ramp and model car activity students experienced virtually. We have used this math test several times in our research and regrettably had no idea that elements of the graphics contributed to misunderstanding.

Unfortunately, our findings suggest that the changes made in response to COVID-19 may have depressed student understanding. Performances on two items (including item 4 in Figure 1) that assessed the main points of the intervention were disappointing compared to results from prior studies. The increase in class size from 3–4 to 14 after COVID and switching to online learning may have reduced the opportunity for repetition and practice. There were reduced opportunities for students to participate in the hands-on activities and participate in conversations about their thinking with other students.

We acknowledge the limitations of this small pilot study to compare knowledge of students when assessed in a pencil and paper format to an oral assessment. We are optimistic about the potential of oral assessments to reveal problem-solving insights of students with math disabilities. The information gained from oral assessment is of value if teachers use it to individualize their instruction. As we learned, oral assessment can also point to areas where graphics or other information are misleading. More research is needed to understand the value of oral assessment despite the increase in time it might add to data collection efforts for students with math disabilities. This experience highlights some of the positive experiences of students learning during COVID-19 virtually at home as well as some of the challenges and risks of reduced outcomes from these virtual learning experiences, especially for students with disabilities.

This blog was written by Sarah Brasiel, program officer for NCSER’s Science, Technology, Engineering, and Math program.

NASA to Kick Off Its Latest National Student Challenge at the 2021 ED Games Expo on June 1

The 8th Annual ED Games Expo will occur next week from June 1 to 5. The free event is all virtual, open to the public, and will showcase game-changing innovations in education technology developed through more than 40 programs at the Department of Education (ED) and across the federal government.

 

NASA National Student Challenge Event at the Ed Expo

One of many noteworthy Expo events will occur on Tuesday, June 1, from 6 to 8 PM Eastern when NASA’s Flight Opportunities program will introduce a new national student challenge. Educators can register to attend this LIVE event on June 1 here. The NASA TechRise Student Challenge will invite teams of sixth- to 12th-grade students to submit ideas for climate or remote sensing experiments to fly on a high-altitude balloon, and space exploration experiments to fly aboard a suborbital rocket.

 

 

NASA developed the NASA TechRise Student Challenge to enable students to have a deeper understanding of Earth’s atmosphere, space exploration, coding and electronics, and a broader understanding of the value of test data. The challenge will also provide students with the opportunity to engage with NASA and technology communities and expose them to careers in science, technology, and space exploration fields.

The challenge will begin accepting applications in August for student teams affiliated with U.S. public, private, and charter schools, including U.S. territories.  The winning teams each will receive $1,500 to build their payloads, as well as an assigned spot on a NASA-sponsored commercial suborbital flight. Balloon flights will offer more than four hours of flight time, while suborbital rockets will provide around three minutes of test time in microgravity conditions. The Flight Opportunities program, based at NASA’s Armstrong Flight Research Center, a part of Space Technology Mission Directorate, is leading the NASA TechRise Student Challenge. The challenge is being administered by California-based Future Engineers, which developed its platform with awards in 2016 and 2017 from the ED/IES SBIR program. Future Engineers’ platform has also been employed to manage past educational and NASA challenges, including the Name the Mars Rover student challenge. 

During the June 1 event, NASA experts will provide information to educators on the official competition. Teachers are invited to join a NASA TechRise Educator Summer Workshop, which will dive into the basics of electronics, coding, and designing for flight. The first workshop will be on Wednesday, July 28, 2021 and repeated on Wednesday, August 11, 2021. For challenge details and to pre-register for the competition, please visit the contest website.

 

More ED Games Expo Events to Engage Students in Hands-On Projects and Challenges

In addition to the NASA event, five more virtual events featuring government programs that engage students in project-based learning will occur on Monday, June 1 between 12:30PM to 6PM Eastern. Topics include students building and flying satellites, programs for museums, local communities and military facilities to engage students in experiential and real-world learning, and a program to inspire students to be inventors and entrepreneurs. See the Expo Agenda here for lineup of events and the ED Games Expo Playlists Page for video trailers by participating developers. 

 

We look forward to "seeing you" at the virtual ED Games Expo starting on June 1!


Edward Metz (Edward.Metz@ed.gov) is the Program Manager for the Small Business Innovation Research program at the US Department of Education’s Institute of Education Sciences.

 

Copy, Paste, Transpose: Math Anxiety Is More Common than You Think

This blog is part of our “What Does This Mean for Me” series and was written by Yuri Lin, a virtual intern for NCER.

 

As an undergraduate student who just completed my required math courses, my days of struggling with math are still fresh in my mind. I know the feelings of shame and anxiety when I struggle to solve math problems, and I know I am far from the only person who has experienced this. I have friends who have blanked on exams and tutored middle schoolers who have experienced the same brand of math anxiety, just a handful of years removed, transposed into different classes.

Math anxiety has been defined as discomfort or nervousness that arises when thinking about doing math or while doing math. In some cases, math anxiety could interfere with one's ability to do math and could lead to lower mathematics achievement. This phenomenon occurs broadly across all age and grade levels, including teachers and adults, and has been estimated to peak in middle and high school. To learn more, I asked four IES-funded researchers to share their discoveries about math anxiety and their advice for students, parents, and math educators.

 

Sian Beilock, PhD (@sianbeilock), is a cognitive scientist and the eighth President of Barnard College at Columbia University. Her research focuses on brain and body factors that affect performance anxiety.

An Exploration of Malleable Social and Cognitive Factors Associated with Early Elementary School Students' Mathematics Achievement

Key Finding: Math anxiety starts early. We focused specifically on children at the start of formal schooling and found that some reported fear and apprehension around math. 

Advice for Parents: For parents, I would stress that it is important not to paint a picture of "some people are good at math and others aren't." We can all get better at math. When parents say things to their children like, "It’s okay; I am not a math person either," even though they are trying to comfort their kids, it sends a very strong signal that some people can do math, and some can't. The result is that kids who are anxious about math avoid it, and an unwanted anxiety-achievement cycle is created.

 

Jeremy Jamieson, PhD, is an Associate Professor of Psychology at the University of Rochester. His research focuses on the physiological and psychological impacts of stress, as well as how to manage stress responses to promote resilience.

Exploring Stress Responses in the Classroom and Reappraising Stress to Facilitate Academic Performance

Key Finding: Math anxiety is not just a psychological problem but also has important consequences for biological functioning. Community college students who reported higher levels of math anxiety also had unhealthy perceptions of stress and lower levels of testosterone (a performance-enhancing hormone) on days when they had to take a math test.

Advice for Students: Feeling stressed and anxious about math shows that you care, and those feelings of stress and anxiety do not mean one is “not good” at math. In fact, you can even use the stress you feel about math to help meet difficult challenges. Your body evolved stress responses to mobilize resources and help you perform. When you believe stress is a tool to help achieve difficult goals, your body will respond with a challenge response (which is like excitement) to assist you in reaching new heights.

 

Leigh McLean, PhD, is an Assistant Research Professor at Arizona State University. Her research focuses on teacher-student interactions in classroom environments and how these interactions affect teacher and student outcomes.

Exploring Elementary Teachers' Feelings, Beliefs, and Effectiveness across Mathematics, Science, and Literacy

Key Finding: When teachers are more math-anxious, so are their students. Importantly, when teachers enjoy teaching math and feel more efficacious in their math teaching, student math anxiety decreases and engagement increases. When teachers and parents have math anxiety, children can pick up on this anxiety, and it can impact both how children feel about math themselves and how they perform in math.

Advice for Math Educators: We would advise anyone who is in a role where they are teaching children math to be aware of their own math-related feelings, especially anxiety. Kids will not only pick up on the content adults teach them but also on the emotional signals adults give off. If a caregiver or teacher is experiencing math anxiety, they could try to find ways to increase their own math enjoyment and confidence, and this would likely benefit children’s learning.

 

Lindsey Richland, PhD (@lerichland), is an Associate Professor at the University of California, Irvine. Her research focuses on children’s development of mathematics thinking and cognitive skills, as well as teacher best practices to support this development.

Drawing Connections to Close Achievement Gaps in Mathematics

Key Finding: State math anxiety, which describes how much anxiety a student feels in a particular situation, changes a lot as students learn to solve problems that require higher order thinking. This suggests that it is not always helpful to make generalizations about trait anxiety, which is believed to be a fairly stable characteristic in individuals.  Instead, it may be more effective to develop specific interventions or learn more about problem types that can affect math anxiety.

Advice for Math Educators and Students: When you’re feeling anxious, you may have worries running through your mind that can distract your attention. One of the best ways to make sure you don’t lose out on learning is to use visual cues to help access information you need. When doing a math problem, write down all your work, rather than trying to do steps in your head. Use prior worked examples to help solve new problems. Teachers can do the same–make sure students have a visual record of classroom instruction that they can return to if their mind wanders or provide worked examples to help students learn new problem-solving techniques. 

 


Written by Yuri Lin, intern for the Institute of Education Sciences’ National Center for Education Research and a Microbiology, Immunology, and Molecular Genetics major at UCLA.

Gender Stereotypes in STEM: Emergence and Prevention

In 2018, Dr. Allison Master and co-PI Andrew Meltzoff were awarded a grant, Gender Stereotypes in STEM: Exploring Developmental Patterns for Prevention. This 4-year project explores how and when gender stereotypes about STEM career pathways emerge. The study also seeks to identify ways to mitigate the effects of such stereotypes, such as whether a growth mindset can lead to changes in student attitudes and outcomes toward STEM. As an undergraduate student majoring in microbiology at UCLA, Yuri Lin, virtual intern at NCER, was interested in learning more about gender inequalities and stereotypes in STEM education. She recently had a chance to talk with Dr. Master about her research and its implications for increasing STEM participation among women.

 

How is American culture affecting the STEM gender gap, and how does the US compare to other countries on this issue?

When children grow up in American culture, they see lots of TV shows and books where mathematicians, scientists, and engineers are men. STEM-based toys are also heavily marketed toward boys rather than girls. Some countries have begun changing the portrayal of gender stereotypes in the media. For example, the UK’s Advertising Standards Authority has recently started banning TV commercials that reinforce gender stereotypes. Some cross-national studies have shown that gender-STEM stereotypes favoring men are linked to women’s lower success and participation in STEM. The United States is one of many Western countries in which women have more equality and freedom to choose their careers but are much less likely to choose STEM careers than men. We still have a lot of work to do in the United States to break down barriers for women in STEM, and we need to focus on helping girls and women see the value in choosing pathways into STEM.

 

Why do you think it is important to examine growth mindset as a potential way to reduce the effects of stereotypes and increase STEM interest in students?

Growth mindsets are beliefs that personal characteristics can be changed, through effort or the right strategies. This is contrasted with fixed mindsets, which are beliefs that those characteristics can’t be changed. Growth mindsets are particularly helpful for struggling students. Students who have a growth mindset remain focused on learning rather than looking smart, believe effort is important, and stay resilient even when they experience setbacks. These attitudes translate into putting forth more effort and determination, which lead to greater success. In our project, we want to know if a growth mindset can help girls stay motivated in computer science, a subject that can have a steep learning curve. Girls in particular often get discouraged when they feel that they don’t have what it takes to succeed in STEM. We hope that teaching girls to have a growth mindset will protect them from these negative stereotypes and increase their confidence in themselves and their sense of belonging in computer science.

 

Considering that your project includes students from grades 1 to 12, how do you plan to share your findings with teachers, students, and policymakers? Are there differences in how you might communicate the information for different age groups?

As a developmental psychologist, I think it’s important to communicate the information about different age groups to everyone! It can be very valuable to frame student motivation in the broader context of how students are growing and changing. Students start to endorse stereotypes about computer science and engineering very early—Grades 1-3—so elementary school is a great time to start counteracting stereotypes by showing a broad representation of who enjoys and succeeds in STEM. We start to see big gender gaps in computer science interest during middle school, so this is a great time to have girls participate in fun and engaging coding classes. And we’ve already noted how important it is for girls in high school to have a growth mindset in their STEM classes.

We have different goals for communicating with teachers, parents, and policymakers. We know that teachers are very busy, so we try to condense things into the most important practical tips. We’ve made short videos and infographics about our research for teachers. For policymakers, we write policy briefs, which combines our research with other findings that are relevant to education policy. And when we talk to parents, we try to focus on the importance of the experiences they provide for their kids. We really value spreading the word about our research to make sure it reaches people who can use it to make a difference. For more information and access to the various resources, please visit the I AM Lab website.

 


Allison Master, PhD (@AllisonMaster), a developmental psychologist and an assistant professor at the University of Houston, has conducted extensive research on the development of motivation and identity in STEM education. 

Written by Yuri Lin (ylin010101@g.ucla.edu), intern for the Institute of Education Sciences and a Microbiology, Immunology, and Molecular Genetics major at UCLA.

"Boys Have It; Girls Have to Work for It": Examining Gender Stereotypes in Mathematics Achievement

In 2020, Andrei Cimpian, along with co-PIs Sapna Cheryan, Joseph Cimpian, and Sarah Lubienski, were awarded a grant for “Boys Have It; Girls Have to Work for It”: The Development and Consequences of Gender Stereotypes About Natural Talent vs. Effort in Mathematics. The goal of this project is threefold: 1) to explore the origins of the gender stereotype that girls achieve in math due to effort and boys achieve in math due to natural talent, 2) to investigate the consequences of these stereotypes, and 3) to identify ways of reducing the negative effects of these stereotypes on mathematics outcomes. In this blog, we interviewed Dr. Andrei Cimpian on his inspiration and insights on this research, as well as his plans to disseminate the findings to education practitioners.

 

Dr. Andrew CimpianWhat spurred your research, and what prior research was foundational for this current study? 

The co-PIs and I were inspired to do this research because we were struck by the contrast between two sets of facts. On the one hand, girls do better in school than boys from kindergarten to grade 12. Women also obtain more bachelor’s and graduate degrees than men. On the other hand, we as a society still think of men as more brilliant and genius-like than women. For example, participants in a 2018 study referred more male than female acquaintances for a job that they were told requires natural smarts1. When the same job was said to require a strong work ethic instead, participants referred equal numbers of women and men.

Of course, societal views of women and men have changed quite a bit over the last century. With respect to general competence, women are now equal with men in the eyes of the American public. But the stereotype that associates “raw,” high-level intellectual talent with men more than women seems to have resisted change. Why?

Our research is testing a promising hypothesis: It is possible that people give different explanations for women’s and men’s intellectual successes, explaining men’s competence as being due primarily to their inborn intellectual talent and women’s as being due to their efforts. This effort-vs.-talent stereotype “explains away” women’s achievements by attributing them to a quality—perseverance—that is less valued in American culture than natural ability.

Versions of this explanatory stereotype have been documented in adults, but our project will provide its first systematic investigation among children. In particular, we will investigate the effort-vs.-talent stereotype in the domain of mathematics because innate ability is particularly valued in this domain1, which might make this stereotype especially consequential.

 

What are some examples of language or behavior that might suggest an individual holds a particular stereotype? Are there potential ways of mitigating the negative effects of stereotypes?

The best example of this stereotype that I can think of—and this is in fact the anecdote that crystallized our team’s interest in this topic—was recounted by co-PI Joseph Cimpian in a recent piece for The Brookings Institution (emphasis is mine):

About five years ago, while Sarah [Lubienski] and I were faculty at the University of Illinois, we gathered a small group of elementary teachers together to help us think through […] how we could intervene on the notion that girls were innately less capable than boys. One of the teachers pulled a stack of papers out of her tote bag, and spreading them on the conference table, said, “Now, I don’t even understand why you’re looking at girls’ math achievement. These are my students’ standardized test scores, and there are absolutely no gender differences. See, the girls can do just as well as the boys if they work hard enough.” Then, without anyone reacting, it was as if a light bulb went on. She gasped and continued, “Oh my gosh, I just did exactly what you said teachers are doing,” which is attributing girls’ success in math to hard work while attributing boys’ success to innate ability. She concluded, “I see now why you’re studying this.”

In terms of what can be done to mitigate the effects of this stereotype, our project will investigate a potential strategy: normalizing effort by making it clear to students that everyone (not just particular groups) needs to work hard to learn math. This message reframes what is viewed as necessary for success in math away from the belief that natural talent is key, thereby undercutting the power of effort-vs.-talent stereotypes.

 

The current study focuses on elementary school students in grades 1 through 4. What was the motivation for choosing this specific age group?

In general, gender stereotypes about intellectual ability seem to emerge quite early. For instance, girls as young as 6 and 7 are less likely than boys to associate being “really, really smart” with members of their own gender. For this reason, we think it is really important to focus on young children—we need to understand when effort-vs.-talent stereotypes first take root!

“Catching” these stereotypes when they first arise is also important for intervention purposes. If left unchecked, the effects of effort-vs.-talent stereotypes may snowball over time (for example, differences in the types of careers that young women and men are motivated to pursue).

 

What plans do you have to disseminate the findings of this research in ways that will be useful for education practitioners? 

We are mindful of the importance of getting this research into the hands of teachers so that they can use it in practice. We hope to write articles on this work for media outlets that draw educationally oriented audiences. To reach parents as well, we will coordinate with popular media outlets to disseminate the results of this work to general audiences. More generally, we will make every effort to ensure that the findings have maximal societal impact, raising awareness of effort-vs.-talent stereotypes among parents, educators, and the general public.

 


Andrei Cimpian, PhD (@AndreiCimpian), Professor of Psychology at New York University, has conducted extensive research on children’s conceptual development, explanations, and motivation in school.

Written by Yuri Lin (ylin010101@g.ucla.edu), intern for the Institute of Education Sciences.

Photo credit: Brian Stauffer


1The full PDF and resources are available at https://www.cimpianlab.com/motivation.