IES Blog

Institute of Education Sciences

Assessing Math Understanding of Students with Disabilities During a Pandemic

For almost two decades, IES/NCSER has funded Brian Bottge and his teams at the University of Kentucky and University of Wisconsin-Madison to develop and test the efficacy of a teaching method called Enhanced Anchored Instruction (EAI), which helps low-achieving middle school students with math disabilities develop their problem-solving skills by solving meaningful problems related to a real-world problem. The research findings support the efficacy of EAI, especially for students with math disabilities. Most recently, Bottge and his team have been researching innovative forms of assessment that more adequately capture what students with disabilities know both conceptually and procedurally in solving math problems. With supplemental funding, IES/NCSER extended Dr. Bottge’s latest grant to test the use of oral assessment to measure student knowledge and compare that with the knowledge demonstrated on a pencil and paper test. The COVID-19 pandemic introduced added challenges to this work when schools closed and students shifted to online education.

Below we share a recent conversation with Dr. Bottge about the experience of conducting research during a pandemic and what he and his team were still able to learn about the value of oral assessment in mathematics for students with disabilities.

What changes did you observe in the intervention implementation by teachers due to the COVID-related shift to online learning?

The shift to online learning created changes in class size and structure. For 38 days (22 days in classroom, 16 days online through a virtual meeting platform), the middle school special education teacher first taught concepts through a widely used video-based anchored problem, the Kim’s Komet episode of the Jasper Project, in which characters compete in a “Grand Pentathlon.” The teacher then engaged the students in a hands-on application of the concepts by running a live Grand Pentathlon. In the Grand Pentathlon, students make their own cars, race them on a full-size ramp, time them at various release points on the ramp, and graph the information to estimate the speed of the cars. The purpose of both units was to help students develop their informal understanding of pre-algebraic concepts such as linear function, line of best fit, variables, rate of change (slope), reliability, and measurement error. Midway through the study, in-person instruction was suspended and moved online. Instead of working with groups of three to four students in the resource room throughout the day, the teacher provided online instruction to 14 students at one time and scheduled one-on-one sessions with students who needed extra help.

What challenges did you observe in the students interacting with the activities and their learning once they shifted to online learning?

All students had access to a computer at home and they were able to use the online platform without much confusion because they had used it in other classes. The screen share feature enabled students to interact with much of the curriculum by viewing the activities, listening to the teacher, and responding to questions, although they could not fully participate in the hands-on part of the lessons. Class attendance and student behavior were unexpectedly positive during the days when students were online. For example, one student had displayed frequent behavioral outbursts in school but became a positive and contributing member of the online class. The ability to mute mics in the platform gave the teacher the option of allowing only one student to talk at a time.

Were students still able to participate in the hands-on activities that are part of the intervention?

For the hands-on activities related to the Grand Pentathlon competition, the teacher taught online and a research staff member manipulated the cars, track, and electronic timers from campus. Students watched their computer screens waiting for their turn to time their cars over the length of the straightaway. The staff member handled each student’s cars and one by one released them from the height on the ramp as indicated by each student. After students had recorded the times, the teacher asked students to calculate and share the speeds of their cars for each time trial height.

Do you have any other observations about the impact of COVID-19 on your intervention implementation?

One of the most interesting observations was parent participation in the lessons. Several parents went beyond simply monitoring how their child was doing during the units to actively working out the problems. Some were surprised by the difficulty level of the math problems. One mother jokingly remarked: I thought the math they were going to do was as easy as 5 + 5 = 10. The next time my son might have to be the parent and I might have to be the student. You all make the kids think and I like that.

When COVID-19 shut down your participating schools, how were you able to adjust your data collection to continue with your research?

We used the same problem-solving test that we have administered in several previous studies (Figure 1 shows two of the items). On Day 1 of the study (pre-COVID), students took the math pretest in their resource rooms with pencil and paper. Due to COVID-19 school closures, we mailed the posttest and test administration instructions to student homes. On the scheduled testing day during an online class session, students removed the test from the envelope and followed directions for answering the test questions while we observed remotely. On Days 2 and 3 of the study (pre-COVID), an oral examiner (OE) pretested individual students in person. The OE asked the student questions, prompting the student to describe the overall problem, identify the information needed for solving the problem, indicate how the information related to their problem-solving plan, and provide an answer. Due to COVID-19, students took the oral posttests online. The teacher set up a breakout room in the platform where the OE conducted the oral assessments and a second member of the research team took notes.

Figure 1. Sample Items from the Problem-Solving Test

During the testing sessions, the OE projected each item on the students’ computer screens. Then she asked the student to read the problem aloud and describe how to solve it. The OE used the same problem-solving prompts as was used on the pretests. For problems that involved graphs or charts, the OE used the editing tools to make notations on the screen as the students directed. One challenge is that oral testing online made it more difficult to monitor behavior and keep students on task. For example, sometimes students became distracted and talked to other people in their house.

What were the results of this study of oral assessment in mathematics for students with disabilities?

Our results suggest that allowing students to describe their understanding of problems in multiple ways yielded depth and detail to their answers. We learned from the oral assessment that most students knew how to transfer the data from the table to an approximate location on the graph; however, there was a lack of precision due to a weak understanding of decimals. For item 4 in Figure 1, the use of decimals confused students who did not have much exposure to decimals prior to or during the study. We also found that graphics that were meant to help students understand the text-based items were in some cases misleading. The representation in item 4 was different than the actual ramp and model car activity students experienced virtually. We have used this math test several times in our research and regrettably had no idea that elements of the graphics contributed to misunderstanding.

Unfortunately, our findings suggest that the changes made in response to COVID-19 may have depressed student understanding. Performances on two items (including item 4 in Figure 1) that assessed the main points of the intervention were disappointing compared to results from prior studies. The increase in class size from 3–4 to 14 after COVID and switching to online learning may have reduced the opportunity for repetition and practice. There were reduced opportunities for students to participate in the hands-on activities and participate in conversations about their thinking with other students.

We acknowledge the limitations of this small pilot study to compare knowledge of students when assessed in a pencil and paper format to an oral assessment. We are optimistic about the potential of oral assessments to reveal problem-solving insights of students with math disabilities. The information gained from oral assessment is of value if teachers use it to individualize their instruction. As we learned, oral assessment can also point to areas where graphics or other information are misleading. More research is needed to understand the value of oral assessment despite the increase in time it might add to data collection efforts for students with math disabilities. This experience highlights some of the positive experiences of students learning during COVID-19 virtually at home as well as some of the challenges and risks of reduced outcomes from these virtual learning experiences, especially for students with disabilities.

This blog was written by Sarah Brasiel, program officer for NCSER’s Science, Technology, Engineering, and Math program.

Two-Year Position in the Institute Of Education Sciences (NCSER) to Support Research on Accelerating Pandemic Recovery for Learners With Disabilities

The National Center for Special Education Research (NCSER) in the Institute of Education Sciences is pleased to announce a two-year position to support the work NCSER is undertaking to accelerate pandemic recovery for learners with disabilities. The position, funded through the American Rescue Plan, is for an Associate Education Research Scientist (position series AD-1730).  The incumbent will work with current NCSER staff to support research that addresses pandemic recovery for students with disabilities and manage projects funded through new pandemic recovery grant competitions and initiatives. 

Those interested in applying can submit their application through USA Jobs through an existing position posting for an Associate Education Research Scientist (AD-1730-00) at https://www.usajobs.gov/GetJob/ViewDetails/603739300.

Please note, IES can support a temporary detail to this position through the Intergovernmental Personnel Act (IPA) Mobility Program. This program provides for the temporary assignment of personnel between the Federal Government and state and local governments, colleges and universities, Indian tribal governments, federally funded research and development centers, and other eligible organizations. More information on an IPA can be found at https://www.opm.gov/policy-data-oversight/hiring-information/intergovernment-personnel-act/#url=Overview.

If you are interested in pursuing this opportunity and have additional questions, contact NCSER Commissioner Joan McLaughlin at NCSER.Commissioner@ed.gov.

NCSER is intending to fill this position as soon as possible, so please apply by August 6, 2021.

Mental Health Awareness Month

The past year and a half have brought new meaning to May’s National Mental Health Awareness Month. As students, families, and school staff navigate virtual, hybrid, and new routines surrounding in-person learning, promoting mental wellbeing has been a large topic of discussion for supporting students and educators.

The National Center for Special Education Research (NCSER) funds projects that include an emphasis on the mental health of students with or at risk for disabilities and their educators. Below are examples of such projects, which focus on supporting students who have internalizing disorders or experienced trauma and preventing and reducing burnout in special education teachers.

Internalizing Disorders

At the University of Connecticut Health Center, Dr. Golda Ginsburg tested the efficacy of a cognitive behavioral therapy (CBT) intervention for youth ages 7-17 in special education with an anxiety disorder as part of the School-based Treatment of Anxiety Research Study (STARS) program. This intervention was designed to be implemented by a school-based mental health clinician and contains seven core modules to help students understand, manage, and cope with anxiety. STARS also includes similarly focused parent training modules. Results demonstrated that parent-reported level of child anxiety decreased after participating in the program. They also suggested that older youth, those with social phobia, and those with more severe anxiety at the start of the study were more likely to benefit from participating in the STARS program.

Dr. Ginsburg has been developing another intervention for anxiety, Teacher Anxiety Program for Elementary Students (TAPES). This is a professional development program that enhances teacher knowledge and skills for identifying and reducing anxiety in students with or without disabilities who have elevated anxiety symptoms. TAPES contains materials informed by CBT to be used class-wide and during teacher-parent-student meetings. The research team will soon be conducting a pilot test of this program.

Additionally, at the University of Wisconsin-Madison, Dr. Stephen Kilgus and colleagues are currently developing the Resilience Education Program (REP), a tier 2 intervention for elementary students at risk for internalizing behaviors. REP consists of three components: A cognitive behavioral instructional curriculum to promote acquisition of social-emotional skills, use of the Check In/Check Out (CICO) system (an existing intervention for promoting the maintenance of acquired skills), and parent skills training that promotes CICO implementation and facilitates positive parent-child relations.

Trauma

At SRI International, Dr. Carl Sumi and colleagues tested the efficacy of the Cognitive Behavioral Intervention for Trauma in Schools (CBITS), a school-based, structured, symptom-focused therapy program for middle school students who have experienced significant trauma and are experiencing related emotional or behavioral challenges. CBITS consists of 10 group therapy sessions and one individual session with a school-based mental health clinician with training on relaxation techniques, cognitive therapy, exposure, and social problem solving. The study found positive impacts of the intervention overall on social-emotional and academic outcomes, but the outcomes varied depending on level of behavioral concerns prior to beginning treatment. Specifically, for youth with more significant externalizing behavior problems (such as aggression), those who experienced CBITS had greater reductions in post-traumatic stress and other emotional and behavioral problems, as well as improved scores on a standardized literacy assessment. In addition, for students with internalizing behavior problems (such as anxiety), participation in CBITS led to better performance on standardized math tests 1 year later.

Teacher Burnout

At Ball State University, Dr. Lisa Ruble and colleagues are adapting an existing manualized intervention for mental health workers, Burnout Reduction: Enhanced Awareness, Tools, handouts, and Education (BREATHE), to be used with special education teachers to reduce burnout. BREATHE is both a prevention and intervention strategy, as it aims to prevent burnout from occurring and reduces burnout when present. Sessions cover CBT stress reduction techniques, meditation and relaxation practices, and social skills training to increase coping skills and the ability to manage stressful job demands.

This blog was written by Alice Bravo, virtual intern for IES and doctoral candidate in special education at the University of Washington, and Jackie Buckley, program officer for NCSER’s Social, Emotional, and Behavioral Competence program. Katie Taylor is the program officer for NCSER’s Educators and School-Based Service Providers program.

Making the Most of a Quarantine Year: Meet the IES Virtual Interns!

April is National Internship Awareness Month, and we want to take this opportunity to highlight the Virtual Student Federal Service (VSFS) internship program that IES has been involved in this year and thank our wonderful interns for their contributions to the National Center for Education Research (NCER) and National Center for Special Education Research (NCSER).

The two IES Centers hired four interns to work on communication and two interns to work on data science. We asked each of them to tell us a little about themselves, their future plans, and what interested them or surprised them about the internship with IES. Here’s what they said.

 

Alice Bravo is pursuing a PhD in special education in the College of Education at the University of Washington.

My research interests keep evolving but are rooted in early intervention for young children with autism spectrum disorder (ASD) using applied behavior analysis and developmental science. Specifically, I am interested in the teaching of imitation and communication skills. In 5 years, I hope to be working as an applied researcher and practitioner, conducting research related to early intervention and ASD while providing training and coaching to caregivers and early intervention/early childhood special education professionals. During my internship with IES, I was really interested in and excited by the breadth of research supported by IES. Reading project abstracts related to virtual reality to support student learning was fascinating! 

Fun fact: I love road trips – I have driven up and down the West Coast and across the country twice! 

 

Bonnie Chan is pursuing a bachelor’s degree in statistics and machine learning at Carnegie Mellon.

I am interested in data science and modeling of data. I am interested in applying these approaches to research in the field of medicine or psychology because it has the most potential to help people and one of the most applicable uses of these approaches. As part of my virtual internship, I have learned how to use PANDAs Python package when cleaning data to prepare to create a visualization of grants funded by NCSER on a U.S. map. In addition, I learned a lot about how grants are funded by the department and the types of projects that are funded. In the future, I would like to pursue a master’s degree in machine learning or other statistical approaches for data science and modeling of data. I think working in the federal government would be a great experience and more rewarding in terms of outcomes than in the public sector or at an institution.

Fun Fact: I really like to dance. I have been dancing since I was 3, so that is 17 years. Right now, I mostly do contemporary dance, but I have done ballet, tap, jazz and other types of dance including competitive dancing in high school. 

 

Chandra Keerthi is pursuing a bachelor’s degree in data science at the Wilfrid Laurier University.

I’m interested in applying statistical models of previous credit ratings to future ones in order to help model human behavior in the area of financial data analysis. I am also really interested in sports analytics, specifically basketball, and in understanding how analytics can help make or sometimes, unintentionally, break teams. In 5 years, I hope to use my skills to help create or innovate a product that will have a positive impact on the world.

Fun fact: I enjoy playing and watching basketball and am a huge fan of sci-fi movies and books (I’m currently reading the first book in the Dune series). In addition, I recently made a program that uses a photo taken from your phone and turns it into 'art' using another art piece (like van Gogh’s The Starry Night) as a reference.

 

Thomas Leonard is pursuing a bachelor’s degree in Economics and Business at Georgetown University.

 

My research interest is in the area of finance. As a virtual intern, I had the opportunity to work on editing and examining abstracts across many different fields of education research, and this has sharpened my technical and analytical skills. In addition, it was interesting to see some of my experiences as a student actually being studied in schools across the country as part of the research that IES funds.

Fun fact: I’m an avid poker player. 

 

 

 

Yuri Lin is pursuing a bachelor’s degree in Microbiology, Immunology, and Molecular Genetics at the University of California, Los Angeles.

I am most interested in cancer genomics, immunology, and psychology. The most surprising detail that I had never thought about before this internship was how government entities like the Department of Education change and are influenced by different presidential administrations. In one of our monthly gatherings, we talked about how each administration has differing visions and values for education, and it struck me that while I saw myself as just a tired college student plinking away at blogs and abstracts in my bedroom, I was actually helping in small ways to fulfill a larger vision for education that sustains across administrations. That was a surprising and rewarding realization to have.

Fun fact: I love music, especially pop music and Russian classical music. There’s so much great music out there, but my favorite would have to be Shostakovich Symphony 5, Movement 4. Nothing feels quite like playing that piece in a huge orchestra with the cymbals crashing, and I hope everyone who hasn’t heard it before can go give it a listen.

 

Shirley Liu is pursuing a bachelor’s degree in English with a double minor in philosophy and data science at Lafayette College.

My research interests are in the areas of communication and data and information science. During this internship, I learned a lot about the human and community aspect of research. I have always viewed research and academia as very solitary fields. They are, but after talking to researchers about the friendships they’ve made in the field, I’ve learned that research is a lot more fruitful (and fun) when you’re doing it with someone whose company you enjoy. I really loved learning about Plain Language Principles! I have already started applying that to my own writing. For example, I am probably the only person in my friend group who knows what nominalization is and why it should be avoided.

Fun fact: My favorite hobby is writing! I have won an undergraduate-level prize for my poetry.

 


In addition to working on abstracts, entering data, creating data visualizations, and helping to update compendia of IES-funded research, our interns have also been busy writing blogs. Here are some recent blogs written by our interns: Autism Awareness & Acceptance Month; What Does This Mean for Me? A Conversation about College and ADHD; and Gender Stereotypes in STEM: Emergence and Prevention.

Autism Awareness & Acceptance Month

April is Autism Awareness and Acceptance Month, a month dedicated to promoting true inclusion of individuals with autism spectrum disorder (ASD) and supporting them in reaching their full potential. In honor of this, we reached out to researchers aiming to improve outcomes for learners with ASD through Early Career Development and Mentoring grants from the National Center for Special Education Research (NCSER). We asked these principal investigators how they got involved in ASD research and about their current NCSER-funded work. Below is what they had to say.

Stephanie Shire, University of Oregon

I first interacted with young children with ASD as a teenage volunteer in a hospital playroom. As I learned more about children with special needs through summer camps and as an in-home aide, I grew more intrigued by the range of strengths and needs of these children. I found joy in finding ways to connect with children who had few or no words, but I lacked the tools to support their growth. This set me on a path to learn about the range of intervention practices and intervention science under the mentorship of Dr. Connie Kasari at the University of California, Los Angeles. My overall goal is to develop and test intervention programs to support the deployment of high-quality practices across the United States and abroad.

In the spirit of this goal, the purpose of my Early Career project, LIFT (Leveraging autism Intervention for Families through Telehealth), is to develop a technology-enabled version of an evidence-based, caregiver-mediated social communication intervention (JASPER; Joint Attention, Symbolic Play, Engagement, and Regulation) to be delivered by community-based early educators serving families of young children with ASD in rural areas. We are currently in Year 1 of our 4-year project. This development year is focused on the creation of the online JASPER intervention and training materials for early intervention and early childhood special education providers. Despite the demands of the pandemic, participating providers have engaged in training using video and role play and the majority are now able to put their skills to use with young children with ASD. We are currently conducting user testing of the online materials and preparing for next year’s randomized controlled trial.

Veronica Fleury, Florida State University

My first experience working with individuals with ASD was in a college course on behavior modification. The professor directed an ASD clinic that provided therapy using many of the strategies we discussed in class. I completed an internship in the clinic and was intrigued by the application of research techniques to promote prosocial behaviors for children with ASD. After college, I secured a full-time research assistantship at the University of Washington in a large ASD study focused on genetics and neurobiology. This was a pivotal experience because I realized this was not the kind of research that I wanted to pursue. The results of these efforts, while extremely valuable, did little to directly improve the lives of the participants. I realized that I wanted to be involved in applied research that allows for quicker uptake by practitioners and benefits for individuals with ASD. In order to be a good applied researcher, I needed practical experience working with children with ASD and their families. Although my entry into preschool special education teaching was initially a means to an end, it drew me in and further fueled my desire to serve children with disabilities. After this experience, I continued my graduate education and am now in an academic position that allows me to use science to address socially significant problems faced by individuals with ASD and their families.

The goal of Project START (Students and Teachers Actively Reading Together), which is part of my Early Career project, is to develop an adaptive shared reading intervention for preschool children with ASD using a sequential, multiple assignment, randomized trial (SMART) design. The results will help determine whether a full-scale efficacy study is worth pursuing for the intervention in its current form or whether additional refinement and testing is necessary.

Melanie Pellecchia, University of Pennsylvania

I became interested in research focused on improving implementation of evidence-based treatments for young children with ASD in under-resourced communities after many years of working with young children with ASD as a behavior analyst overseeing publicly funded early intervention programs. While working within a large, urban public-service system, I observed the widespread disparities in access to high-quality intervention and challenges with implementing evidence-based interventions to scale for young children with ASD. I sought to pursue an academic research career focused on improving these implementation challenges.

As part of my Early Career project, I am iteratively developing a toolkit of implementation strategies designed to improve parent coaching for young children with ASD in Part C early intervention systems. I am currently in my third year of this project and am incorporating information learned from a variety of sources to develop the toolkit, including direct observations of early intervention sessions, qualitative interviews identifying barriers and facilitators to using parent coaching within early intervention, literature on best practices in parent coaching and parent-mediated interventions for young children with ASD, and feedback from expert and community advisory panels. The toolkit will include a series of infographics, videos, and a community of practice housed within an online platform. This year I plan to conduct a pilot study of the toolkit to assess its feasibility and promise for improving the use of parent coaching for young children with ASD in Part C service systems.

Marisa Fisher, Michigan State University

Most people assume I got into the field because I grew up with an older brother with Williams Syndrome. But I didn't really think of myself as a sibling of a person with a disability and how that experience had shaped my life until I was in graduate school. The real reason I entered the field was because of three little boys with ASD with whom I worked as a behavior therapist when I was in college. What was originally a job became a passion for supporting people with ASD and other disabilities and finding better ways to teach skills and improve outcomes. I knew I wanted to go to graduate school, and it was my experience with these boys and my work at an ASD research lab that pushed me to pursue a doctorate in special education so that I could continue to work with people with disabilities.

Through my work with individuals with ASD, I began to realize the social struggles they and my brother experienced and became interested in studying experiences of social victimization and finding out why people with disabilities are more socially vulnerable than individuals without disabilities. The goal of my Early Career project is to do just that. A key part of this project involves assessing students’ self-reported bullying experiences. Although my original plan was to adapt and expand on existing measures, this didn’t result in a feasible assessment. Therefore, I turned my attention toward developing and testing an assessment that was appropriate for students with ASD and plan to use it to better understand the risk factors and consequences of bullying for these students. In general, my research is evolving from identifying and describing the risk factors to developing interventions to address those risk factors and reduce experiences of social victimization. My approach is to teach individuals with ASD to recognize and respond to situations and to evaluate ways to change attitudes toward individuals with ASD and improve social inclusion.

This blog was written by Alice Bravo, virtual intern for IES and doctoral candidate in special education at the University of Washington, and Katie Taylor, program officer for NCSER’s Early Career Development and Mentoring program.