Inside IES Research

Several efforts around the country are re-examining the skills students need to be prepared for the 21^st century. Frontier digital technologies such as artificial intelligence, quantum computing, and blockchain carry the potential—and in some cases have already begun—to radically transform the economy and the workplace. Global engagement and national competitiveness will likely rely upon the skills, deep understanding, and leadership in these areas.

These technologies run on a new type of fuel: data, and very large amounts of it. The “big data” revolution has already changed the way modern businesses, government, and research is conducted, generating new information and shaping critical decisions at all levels. The volume and complexity of modern data has evolved to such a degree that an entire field—data science—has emerged to meet the needs of these new technologies and the stakeholders employing them, drawing upon an inter-disciplinary intersection of statistics, computer science, and domain knowledge. Data science professionals work in a variety of industries, and data now run many of the systems we interact with in our daily life—whether smart voice assistants on our phone, social media platforms in our personal and civic lives, or Internet of Things infrastructure in our built environment.

Students in grades K-12 also interact with these systems. Despite the vast amount of data that students are informally exposed to, there are currently limited formal learning opportunities for students to learn how to understand, assess, and work with the data that they encounter in a variety of contexts. Data science education in K-12 is not widespread, suggesting that our education system has not invested in building capacity around these new and important skill sets. A review of the NCES 2019 NAEP High School Transcript Study (HSTS) data revealed that only 0.07% of high school graduates took a data science course, and 0.04% of high school graduates took an applied or interdisciplinary data science course in health informatics, business, energy, or other field. Critically, education research informing the design, implementation, and teaching of these programs is similarly limited.

To develop a better understanding of the state of data science education research, on October 26, 2021, NCER convened a Technical Working Group (TWG) panel to provide recommendations to NCER on 1) the goals for K-12 data science education research, 2) how to improve K-12 data science education practice, 3) how to ensure access to and equity in data science education, and 4) what is needed to build an evidence base and research capacity for the new field. The five key recommendations from the panel are summarized in a new report.  

• Recommendation 1. Articulate the Developmental Pathway—Panelists recommended more research to better articulate K-12 learning pathways for students.
• Recommendation 2: Assess and Improve Data Science Software—Panelists suggested additional research to assess which data analysis software tools (tinker-based tools, spreadsheets, professional software, or other tools) should be incorporated into instruction and when, in order to be developmentally appropriate and accessible to all learners.
• Recommendation 3: Build Tools for Measurement and Assessment—Panelists advocated for additional research to develop classroom assessment tools to support teachers and to track student success and progress, and to ensure students may earn transferable credit for their work from K-12 to postsecondary education.
• Recommendation 4: Integrate Equity into Schooling and Systems—Panelists emphasized the importance of equity in opportunities and access to high quality data science education for all learners. Data science education research should be conducted with an equity lens that critically examines what is researched and for whom the research benefits.
• Recommendation 5: Improve Implementation—Panelists highlighted several systematic barriers to successfully implementing and scaling data science education policies and practices, including insufficient resources, lack of teacher training, and misalignment in required coursework and credentials between K-12, postsecondary education, and industry. The panel called for research to evaluate different implementation approaches to reduce these barriers and increase the scalability of data science education policies and practices. 

Given the limited evidence base informing data science education at the K-12 level, panelists expressed a sense of urgency for additional research, and for expanded research efforts to quickly build an evidence base to evaluate the promise of, practices for, and best ways to impart data science education. These transformations may carry significant implications for career and technical skills, online social and civic engagement, and global citizenship in the digital sphere.   

Importantly, this report highlights more research is still needed—and soon. IES looks forward to the field’s ideas for research projects that address what works, for whom, and under which conditions within data science education and will continue to engage the education research community to draw attention to critical research gaps in this area.

Written by Zarek Drozda, 2021-2022 FAS Data Science Education Impact Fellow.

Each generation faces its own societal challenges. Two prominent issues—the climate crisis and America’s political divide—are heavy burdens for today’s youth. Without explicit focus in schools, it is hard to imagine how children will learn to work across differences and collaborate with others to solve complex environmental problems. Youth are very capable people, and school comes alive when they feel agency and see how their efforts matter in the community. Service-learning can help teachers make instruction feel relevant and teach skills that lead to civic engagement as youth learn to design, implement, and evaluate solutions to problems that are important to them. In this interview blog, the Connect Science project team explains how they developed curriculum and professional development to support teachers to engage their students in service-learning experiences.

Can you tell us about Connect Science and what it looks like in action?

Fueled by an IES Development and Innovation grant, our team developed and evaluated a science-based service-learning approach for the upper-elementary school years. In doing so, we answered a need that teachers and schools face as they strive to create engaging experiences aligned with the Next Generation Science Standards (NGSS).

Connect Science is a 12-week project-based learning unit for upper elementary students. Early on, teachers and students explore topics of energy and natural resources using lessons aligned with the NGSS. Teachers guide student learning on what it means to be an engaged citizen and on the social and collaborative skills needed to take action in the community. To prepare, teachers receive five days of professional development and follow-up coaching. Teachers also receive a Connect Science manual, related books, and science materials.

But what does Connect Science actually look like in action? Imagine fourth graders engaged in a science unit on renewable and non-renewable resources. The students learn about different energy sources and then discuss pros and cons of each source. They become aware that non-renewable energy resources are rapidly diminishing and would not always be available to generate electricity. The awareness of this problem energizes them to promote energy conservation. Toward that goal, the students decide to educate other students and families at their school about energy use. At the next open house night, they turn their cafeteria into an energy fair where they share important information. For example, one group of students teaches about what types of energy sources were used in their state to produce electricity and another group teaches ways that people can save energy in their home. Before and after the energy fair, the students administer a pre- and post-survey on energy facts to size up what their visitors learned.

How did the IES grant support the development and pilot testing of Connect Science?

In the first two years of this grant, we developed and tested materials with teachers. In the third year, we conducted a randomized controlled trial of Connect Science involving 41 classrooms with 20 in Connect Science and 21 in a waitlist comparison group, resulting in a student sample of 868 students (423 students participated in the intervention).

We found that Connect Science impacted teacher practices and student outcomes. Teachers in the Connect Science group were more effective at engaging in the two NGSS practices that we measured: eliciting and building on prior knowledge and creating opportunities for student critique, explanation, and argument. Further, we saw higher science achievement and energy attitudes and behaviors in the intervention than control condition. The social skill results hinged on the fidelity of implementation. When teachers used more Connect Science practices, students showed improved communication and social competence. As a result of these findings, Connect Science is designated as a Promising Program by the Collaborative for Academic, Social, and Emotional Learning (CASEL).

What are the implications of your findings?

Too few projects integrate academic and social learning in schools. Often, high-quality NGSS materials are developed with little thought about the social skills students need to engage in that instruction. Likewise, social and emotional learning is often taught separately from academic content. Service-learning is a framework that bridges these two areas and allows students to engage in authentic, science-based work. Given our experiences, we have a few recommendations for educators eager to use service-learning.

• Teach social, emotional, and collaborative skills with intention before launching into group work. In the elementary schools, children thrive from being in supportive caring classrooms and they respond well to lessons on active listening, respectful communication, and understanding people with multiple perspectives.
• Leverage the existing curriculum and build in service-learning experiences. Rather than adding one more new topic, look at existing curricular topics and use service-learning to facilitate deep learning on content areas that already part of the curriculum.
• Amplify youth voice. Teachers need to work with students to identify a relevant community problem and generate solutions to that problem. We carefully developed the Connect Science materials to be more teacher-directed toward the beginning of the unit and more student-directed toward the end. This approach was based both on theoretical and empirical work supporting the importance of student autonomy.

Sara Rimm-Kaufman is the Commonwealth Professor of Education at the University of Virginia School of Education and Human Development. Her recent book for teachers, SEL from the Start, is based on the Connect Science work.

Eileen Merritt is a Research Scientist in the College of Natural Resources and the Environment at Virginia Tech. Her research and teaching focus on environmental and sustainability education.

Tracy Harkins is the owner of Harkins Consulting, LLC in Maine. Her focus is providing professional development and resources to engage and motivate student learners through service-learning. She will be offering an upcoming Connect Science Institute in Summer 2022.

For questions about this project, please contact Corinne.Alfeld@ed.gov, NCER program officer.

As part of our recognition of National Disability Employment Awareness Month (NDEAM), we asked IES-funded researcher L. Penny Rosenblum how having a disability impacted the development of her career as a special education researcher.

As a person with a visual impairment, how have your background and experiences shaped your scholarship and career?

I have a congenital visual impairment, so I have had low vision all my life. When I began my undergraduate studies, I quickly realized that I wanted to become a teacher of students with visual impairments (TVI). Once I began work, I came to the realization that I could have a larger and more sustaining impact on the education of students with visual impairments if I prepared TVIs. After earning my doctorate, I first was faculty at Florida State University and then at the University of Arizona. The combination of my own experiences as a child and adult with a visual impairment coupled with my experiences teaching children and then preparing TVIs worked together to shape my research agenda.

What got you interested in a career in special education research?

During my master’s program at Peabody College of Vanderbilt University, I was hired to enter data for a research study. I saw a pattern in the data others had not noticed and I shared this observation with the lead researcher and his doctoral students. This was a pivotal moment for me and sparked an interest in research. When I began my doctoral program and started to learn more about research methods and how outcomes can be used to shape intervention and policy, I was hooked!

What has been the biggest challenge you have encountered and how did you overcome the challenge?

As a researcher, the biggest challenge is funding. I was funded by “soft money” (funding through external sources) at the University of Arizona for 2 decades, the last 7.5 of which were primarily with funding from NCSER. During my career in academia, colleagues and I spent countless hours writing grants. I wish there were more efficient mechanisms to fund research so that researchers can spend more time engaged in research and less time chasing dollars to do research.

How does your research contribute to a better understanding of how to support students with disabilities?

I engage in research that directly impacts students with visual impairments. I was privileged to serve as a project director for two related NCSER projects: AnimalWatch-VI Suite: A comprehensive program for increasing access to science and math for students with visual impairments and An Intervention to Provide Youth with Visual Impairments with Strategies to Access Graphical Information in Math Word Problems. Through these projects we developed materials to support students at the middle school math level to build their skills with the ultimate goal of having more students with visual impairments enter STEM careers. More specifically, the first project developed and tested an instructional program that teaches students with visual impairment computation, fractions, and variables and expressions through solving math word problems embedded in an environmental science context; the second one developed and tested a program to teach students to locate and understand information in graphics that accompany math problems using tactile graphics and accessible image descriptions. I am proud that the materials we developed are available through the American Printing House for the Blind. Our two apps are available at no cost!

• AnimalWatch Vi Suite
• AnimalWatch Vi: Building Graphics Literacy

In your area of research, what do you see as the greatest research needs or recommendations to improve the career outcomes of students with disabilities?

We live in a digital world and until we have addressed the issue of universal access, students with visual impairments will continue to be at a disadvantage. If you’re at a disadvantage in K-12 education, then you’re not going to be as well prepared as others for post-secondary education and employment. I’d like to see research funding that addresses access issues and the development of technologies and tools to level the playing field for all students.

How can the broader education research community better support the careers and scholarship of researchers with disabilities?

Mentorship is so important to me. I have been fortunate in my journey to have some amazing mentors, including Dr. Carole R. Beal who was a principal investigator on the two NCSER-funded projects described above. Dr. Beal was always willing to discuss accommodations I needed due to my visual impairment and to work with me to find solutions. She mentored me in research methodology and professional writing. Researchers, whether they have a disability or not, need to mentor the next generation. I think this is even more important if an emerging scholar has a disability or is from another marginalized group.

What advice would you give to emerging scholars with disabilities who are pursuing a career in education research?

When I think about advice, I again immediately go back to mentorship. I encourage emerging scholars to seek out mentors, both with and without disabilities and in and outside their professional field. I also think it is important to seek out and take advantage of opportunities that come your way, and not wait for someone to come to you. The more networking you can do, the more doors that will open for you. If you’re passionate about your field and your work, people will quickly look beyond your disability and focus on your commitment and skills as a researcher.

L. Penny Rosenblum, PhD is the owner of Vision for Independence, LLC. She has more than 35 years of experience in the field of visual impairment.

This year, Inside IES Research is publishing a series of interviews (see here and here) showcasing a diverse group of IES-funded education researchers and fellows that are making significant contributions to education research, policy, and practice. This NDEAM blog post was produced by Katina Stapleton (Katina.Stapleton@ed.gov), co-Chair of the IES Diversity and Inclusion Council and NCER Program Officer, and Amy Sussman (Amy.Sussman@ed.gov), NCSER Program Officer. See this related NDEAM blog post by NCSER Program Officer Akilah Swinton Nelson (Akilah.Nelson@ed.gov) for information about IES Research on improving career readiness and employment outcomes for students with disabilities.

IES funds cutting-edge researchers who often bring multiple disciplines together. Dr. Maithilee Kunda (Vanderbilt University) is one such researcher who stands at the juncture of multiple fields, using artificial intelligence (AI) to address questions related to cognition and autism spectrum disorder. Recently, Dr. Kunda received an award from the National Center for Special Education Research to develop an educational game that leverages AI to help students with autism spectrum disorder better infer and understand the beliefs, desires, and emotions of others. As a computer scientist and woman of color performing education research, Dr. Kunda exemplifies the value that diverse backgrounds, experiences, and disciplines bring to the field.

Bennett Lunn, a Truman-Albright Fellow at IES, asked Dr. Kunda about her work and background. Her responses are below.

As a woman of color, how have your background and experiences shaped your scholarship and career?

In college, I was a math major on the theory track, which meant that my math classes were really hard! I had been what one might call a “quick study” in high school, so it was a new experience for me to be floating around the bottom quartile of each class. The classes were mostly men, but it happened that there was a woman of color in our cohort—an international student from Colombia—and she was flat-out brilliant. She would ask the professor a question that no one else even understood, but the professor’s eyes would light up, and the two of them would start having some animated and incomprehensible discussion about whatever “mathy” thing it was. That student’s presence bestowed upon me a valuable gift: the ability to assume, without even thinking twice, that women of color quite naturally belong in math and science, even at the top of the heap! I don’t even remember her name, but I wish I could shake her hand. She was a role model for me and for every other student in those classes just by being who she was and doing what she did.

I have been extremely lucky to have seen diverse scientists and academics frequently throughout my career. My very first computer science teacher in high school was a woman. At a high school science camp, my engineering professor was a man who walked with two forearm crutches. Several of my college professors in math, chemistry, and robotics were women. My favorite teaching assistant in a robotics class was a Black man. In graduate school, I remember professors and senior students who were women, LGBTQ people, and people of color. Unfortunately, I know that the vast majority of students do not have access to such a wealth of diverse role models. It is heartening, though, that even a single role model—just by showing up—has so much power to positively shape the perceptions of everyone who sees them in their rightful place, be it in STEM, academia, or whatever context they inhabit.

What got you interested in a career in education science?

I read a lot of science fiction and fantasy growing up, and in high school, I was wrestling with why I liked these genres so much. I came up with a pet theory about fiction writing. All works of fiction are like extended thought experiments; the author sets up some initial conditions—characters, setting, etc.—and they run the experiment via writing about it. In general fiction, the experiments mostly involve variables at the people scale. In sci-fi and fantasy, on the other hand, authors are trying to run experiments at civilization or planetary scales, and that’s why they have to create whole new worlds to write about. I realized that was why I loved those genres so much: they allowed me to think about planetary-scale experiments! 

This “what if” mindset has continued to weave itself throughout my scholarship and career.

How did it ever become possible for humans to imagine things that don’t exist? Why do some people think differently from others, and how can we redesign the workings of our societies to make sure that everyone is supported, enriched, and empowered to contribute to their fullest potential? These kinds of questions fuel my scientific passions and have led me to pursue a variety of research directions on visual thinking, autism, AI, and education.

How does your research contribute to a better understanding of the importance of neurodiversity and inclusion in education?

Early in graduate school, and long before I heard the term neurodiversity, the first big paper I wrote was a re-analysis of several research studies on cognition in autism. This research taught me there can be significant individual variation in how people think. Even if 99 other people with similar demographic characteristics happen to solve a problem one particular way, that does not mean that the hundredth person from the same group is also going to solve the problem that way.

I realized much later that this research fits very well into the idea of neurodiversity, which essentially observes that atypical patterns of thinking should be viewed more as differences than as being inherently wrong or inadequate. Like any individual characteristics you have, the way you think brings with it a particular set of strengths and weaknesses, and different kinds of thinking come with different strengths and weaknesses.

Much of my team’s current research is a continuation of this theme. For example, in one project, we are developing new methods for assessing spatial skills that dig down into the processes people use to solve problems. This view of individual differences is probably one that teachers know intuitively from working one-on-one with students. One of the challenges for today’s education research is to continue to bring this kind of intuitive expertise into our research studies to describe individual differences more systematically across diverse learner populations.

In your area of research, what do you see as the greatest research needs or recommendations to address diversity and equity and improve the relevance of education research for diverse communities of students and families?

For the past 3 years, I have been leading an IES project to create a new educational game called Film Detective to help students with autism spectrum disorder improve their theory of mind (ability to take another’s perspective) and social reasoning skills. This was my first experience doing research on an interactive application of this kind. I was a newcomer to the idea of participatory design, which basically means that instead of just designing for some particular group of users, you bring their voices in as active contributors early in the design process. Our amazing postdoc Dr. Roxanne Rashedi put together a series of early studies using participatory methods, so we had the opportunity to hear directly from middle schoolers on the spectrum, their parents, and their teachers about what they needed and wanted to see in this kind of technology.

In one of these studies, we had students try out a similar education game and then give us feedback. One young man, about 11 or 12 years old, got frustrated in the middle of the session and had a bit of a meltdown. After he calmed down, we asked him about the game and what he would like to see taught in similar games. He told us that he would really like some help in learning how to handle his frustration better so that he could avoid having those kinds of meltdowns. Impressed by his self-awareness and courage in talking to us about his personal challenges, we ended up designing a whole new area in our game called the Relaxatron arcade. This is where students can play mini-games that help them learn about strategies for self-regulation, like deep breathing or meditation. This whole experience reinforced for me the mindset of participatory design: we are all on a team—researchers, students, parents, and teachers—working collaboratively to find new solutions for education.

We are also proud to work with Vanderbilt’s Frist Center for Autism and Innovation to make our research more inclusive and participatory. One of the many excellent programs run by this center is a software internship program for college students or recent graduates on the spectrum. This summer, we are pleased to be welcoming three Frist Center interns who will be helping us on our Film Detective project.

What has been the biggest challenge you have encountered and how did you overcome the challenge?

Throughout my career, I seem to have gravitated towards questions that not many other people are asking, using methods that not many other people are using. For example, I am a computer scientist who studies autism. My research investigates visual thinking, but not vision. I work in AI, but mostly in areas out of the mainstream.

I get a lot of personal and intellectual satisfaction out of my research, but I do face some steep challenges that I believe are common for researchers working in not-so-mainstream areas. For instance, it is sometimes harder to get our papers published in the big AI conferences because our work does not always follow standard patterns for how studies are designed and implemented. And I do experience my share of impostor syndrome (feeling unqualified for your job even when you are performing well) and FOMO (fear of missing out), especially when I come across some trendy paper that already has a thousand citations in 3 months and I think to myself, “Why am I not doing that? Should I be doing that?”

I try to remember to apply the very lessons that my research has produced, and I am fortunate to have friends and colleagues who help lift me out of self-doubt. I actively remind myself about the importance to our species of having diverse forms of thinking and how my own individual view of things is a culmination of my unique lifetime of educational and intellectual experiences. That particular perspective—my perspective—is irreplaceable, and, more than any one paper or grant or citation, it is the true value I bring to the world as a scientist.

How can the broader education research community better support the careers and scholarship of researchers from underrepresented groups?

I think research communities in general need to recognize that inclusion and diversity are everybody’s business, regardless of what someone’s specific research topic is. For example, we assume that every grant proposal and paper follow principles of rigorous and ethical research design, no matter the specific methodology. While some researchers in every discipline specialize in thinking about research design from a scholarly perspective, everyone has a baseline responsibility for knowing about it and for doing it.

Similarly, while we will always want and need researchers who specialize in research on inclusion and diversity, these topics should not be considered somehow peripheral to “real science." They are just as much core parts of a discipline as anything else is. As I constantly remind my students, science is a social enterprise! The pool of individual minds that make our discoveries for us is just as important as any piece of equipment or research method.

What advice would you give to emerging scholars from underrepresented, minoritized groups that are pursuing a career in education research?

A few years ago, when I was a newly minted assistant professor, I went to a rather specialized AI symposium where I found myself to be one of only two women there—out of over 70 attendees! The other woman was a senior researcher whom I had long admired but never met, and I felt a bit star-struck at the idea of meeting her. During one of the coffee breaks, I saw her determinedly heading my way. I said to myself as she approached, “Be cool, Maithilee, be cool, don’t mention the women thing…”  I was gearing myself up to have a properly research-focused discussion, but when she arrived, the very first words out of her mouth were, “So, there’s only the two of us, huh!” We both burst out laughing, and over the next couple of days, we talked about our research as well as about the lack of diversity at the symposium and in the research area more broadly.

The lesson I learned from this wonderful role model was that taking your rightful place in the research community does not mean papering over who you are. Certain researchers are going to be rarities, at least for a while, because of aspects of who we are, but that is nothing to hide. The value we bring as scientists comes from our whole selves and we should not just accept that but embrace and celebrate it.

This blog is part of a series of interviews showcasing a diverse group of IES-funded education researchers that are making significant contributions to education research, policy, and practice. For the first blog in the series, please see Representation Matters: Exploring the Role of Gender and Race on Educational Outcomes.

Dr. Maithilee Kunda is the director of the Laboratory for Artificial Intelligence and Visual Analogical Systems and founding investigator for the Frist Center for Autism and Innovation at Vanderbilt University. This interview was produced and edited by Bennett Lunn, Truman-Albright Fellow for the National Center for Education Research and the National Center for Special Education Research.

For almost two decades, IES/NCSER has funded Brian Bottge and his teams at the University of Kentucky and University of Wisconsin-Madison to develop and test the efficacy of a teaching method called Enhanced Anchored Instruction (EAI), which helps low-achieving middle school students with math disabilities develop their problem-solving skills by solving meaningful problems related to a real-world problem. The research findings support the efficacy of EAI, especially for students with math disabilities. Most recently, Bottge and his team have been researching innovative forms of assessment that more adequately capture what students with disabilities know both conceptually and procedurally in solving math problems. With supplemental funding, IES/NCSER extended Dr. Bottge’s latest grant to test the use of oral assessment to measure student knowledge and compare that with the knowledge demonstrated on a pencil and paper test. The COVID-19 pandemic introduced added challenges to this work when schools closed and students shifted to online education.

Below we share a recent conversation with Dr. Bottge about the experience of conducting research during a pandemic and what he and his team were still able to learn about the value of oral assessment in mathematics for students with disabilities.

What changes did you observe in the intervention implementation by teachers due to the COVID-related shift to online learning?

The shift to online learning created changes in class size and structure. For 38 days (22 days in classroom, 16 days online through a virtual meeting platform), the middle school special education teacher first taught concepts through a widely used video-based anchored problem, the Kim’s Komet episode of the Jasper Project, in which characters compete in a “Grand Pentathlon.” The teacher then engaged the students in a hands-on application of the concepts by running a live Grand Pentathlon. In the Grand Pentathlon, students make their own cars, race them on a full-size ramp, time them at various release points on the ramp, and graph the information to estimate the speed of the cars. The purpose of both units was to help students develop their informal understanding of pre-algebraic concepts such as linear function, line of best fit, variables, rate of change (slope), reliability, and measurement error. Midway through the study, in-person instruction was suspended and moved online. Instead of working with groups of three to four students in the resource room throughout the day, the teacher provided online instruction to 14 students at one time and scheduled one-on-one sessions with students who needed extra help.

What challenges did you observe in the students interacting with the activities and their learning once they shifted to online learning?

All students had access to a computer at home and they were able to use the online platform without much confusion because they had used it in other classes. The screen share feature enabled students to interact with much of the curriculum by viewing the activities, listening to the teacher, and responding to questions, although they could not fully participate in the hands-on part of the lessons. Class attendance and student behavior were unexpectedly positive during the days when students were online. For example, one student had displayed frequent behavioral outbursts in school but became a positive and contributing member of the online class. The ability to mute mics in the platform gave the teacher the option of allowing only one student to talk at a time.

Were students still able to participate in the hands-on activities that are part of the intervention?

For the hands-on activities related to the Grand Pentathlon competition, the teacher taught online and a research staff member manipulated the cars, track, and electronic timers from campus. Students watched their computer screens waiting for their turn to time their cars over the length of the straightaway. The staff member handled each student’s cars and one by one released them from the height on the ramp as indicated by each student. After students had recorded the times, the teacher asked students to calculate and share the speeds of their cars for each time trial height.

Do you have any other observations about the impact of COVID-19 on your intervention implementation?

One of the most interesting observations was parent participation in the lessons. Several parents went beyond simply monitoring how their child was doing during the units to actively working out the problems. Some were surprised by the difficulty level of the math problems. One mother jokingly remarked: I thought the math they were going to do was as easy as 5 + 5 = 10. The next time my son might have to be the parent and I might have to be the student. You all make the kids think and I like that.

When COVID-19 shut down your participating schools, how were you able to adjust your data collection to continue with your research?

We used the same problem-solving test that we have administered in several previous studies (Figure 1 shows two of the items). On Day 1 of the study (pre-COVID), students took the math pretest in their resource rooms with pencil and paper. Due to COVID-19 school closures, we mailed the posttest and test administration instructions to student homes. On the scheduled testing day during an online class session, students removed the test from the envelope and followed directions for answering the test questions while we observed remotely. On Days 2 and 3 of the study (pre-COVID), an oral examiner (OE) pretested individual students in person. The OE asked the student questions, prompting the student to describe the overall problem, identify the information needed for solving the problem, indicate how the information related to their problem-solving plan, and provide an answer. Due to COVID-19, students took the oral posttests online. The teacher set up a breakout room in the platform where the OE conducted the oral assessments and a second member of the research team took notes.

Figure 1. Sample Items from the Problem-Solving Test

During the testing sessions, the OE projected each item on the students’ computer screens. Then she asked the student to read the problem aloud and describe how to solve it. The OE used the same problem-solving prompts as was used on the pretests. For problems that involved graphs or charts, the OE used the editing tools to make notations on the screen as the students directed. One challenge is that oral testing online made it more difficult to monitor behavior and keep students on task. For example, sometimes students became distracted and talked to other people in their house.

What were the results of this study of oral assessment in mathematics for students with disabilities?

Our results suggest that allowing students to describe their understanding of problems in multiple ways yielded depth and detail to their answers. We learned from the oral assessment that most students knew how to transfer the data from the table to an approximate location on the graph; however, there was a lack of precision due to a weak understanding of decimals. For item 4 in Figure 1, the use of decimals confused students who did not have much exposure to decimals prior to or during the study. We also found that graphics that were meant to help students understand the text-based items were in some cases misleading. The representation in item 4 was different than the actual ramp and model car activity students experienced virtually. We have used this math test several times in our research and regrettably had no idea that elements of the graphics contributed to misunderstanding.

Unfortunately, our findings suggest that the changes made in response to COVID-19 may have depressed student understanding. Performances on two items (including item 4 in Figure 1) that assessed the main points of the intervention were disappointing compared to results from prior studies. The increase in class size from 3–4 to 14 after COVID and switching to online learning may have reduced the opportunity for repetition and practice. There were reduced opportunities for students to participate in the hands-on activities and participate in conversations about their thinking with other students.

We acknowledge the limitations of this small pilot study to compare knowledge of students when assessed in a pencil and paper format to an oral assessment. We are optimistic about the potential of oral assessments to reveal problem-solving insights of students with math disabilities. The information gained from oral assessment is of value if teachers use it to individualize their instruction. As we learned, oral assessment can also point to areas where graphics or other information are misleading. More research is needed to understand the value of oral assessment despite the increase in time it might add to data collection efforts for students with math disabilities. This experience highlights some of the positive experiences of students learning during COVID-19 virtually at home as well as some of the challenges and risks of reduced outcomes from these virtual learning experiences, especially for students with disabilities.

This blog was written by Sarah Brasiel, program officer for NCSER’s Science, Technology, Engineering, and Math program.