The National Assessment of Educational Progress (NAEP) is the largest nationally representative and ongoing assessment of subject knowledge among students in public and private schools in the United States. On the 2017 eighth grade mathematics assessment, 38% of students without disabilities scored at the NAEP Proficient level or above while 25% scored below the NAEP Basic level. However, for students with disabilities, math achievement levels were much worse. Only about 9% of students with disabilities scored at the NAEP Proficient level or above whereas 69% scored below the NAEP Basic level. In response to this gap, in 2021, the National Center for Special Education Research (NCSER) released a funding opportunity to coincide with the release of the 2017 Grade 8 NAEP Mathematics response process data. NCSER intended to support research that explores how learners with disabilities interact with the NAEP digital assessment to better support these learners in test-taking environments and determine whether and how that information could be used to inform instructional practices. There is much to learn from research on NAEP process data for understanding test-taking behaviors and achievement of learners with disabilities. Below we showcase the latest findings from currently funded research and encourage more investigators to conduct research with newly released process data.
Since 2017, administrations of NAEP have captured a variety of response process data, including keystrokes as learners progress through the assessment, how learners use the available tools (such as the calculator), and how accommodations (for example, text-to-speech or more time to complete the assessment) affect performance. Besides score data, NAEP datasets also include survey data from learners, teachers, and schools, and information on test item characteristics and student demographics (including disability). Together, these data provide a unique opportunity for researchers to conduct an in-depth investigation of the test-taking behavior and the mathematics competencies of learners with disabilities compared to their peers without disabilities.
In July 2021, IES awarded two grants to conduct research using NAEP process data. The results of these projects are expected to improve the future development and administration of digital learning assessments, identify needed enhancements to mathematics instruction, and highlight areas where further research is needed. Although these projects are ongoing, we would like to highlight findings from one of the funded projects awarded to SRI International and led by principal investigator Xin Wei entitled Analysis of NAEP Mathematics Process, Outcome, and Survey Data to Understand Test-Taking Behavior and Mathematics Performance of Learners with Disabilities.
The findings from this study, recently published in Autism, is an example of the power of process data to shed new light on learners with disabilities. Focusing on autistic students, Xin Wei and her team analyzed data from 15 items on the NAEP math assessment, their response time in seconds, their score on the items (including partially correct scoring), and survey data related to their enjoyment, interest, and persistence in math. They also analyzed the content of each item using Flesch Reading Ease scores to measure the reading difficulty level of the item. Finally, they rated each item based on the complexity of any social context of the item, as prior research has shown that these contexts can be more challenging for autistic students. They conducted statistical analyses to compare the performance of autistic students with extended time accommodations, autistic students without accommodations, and general education peers. The researchers were not only looking for any areas of weakness, but also areas of strength. Previous studies have demonstrated that autistic people frequently excel in abstract spatial reasoning and calculation tasks, relying more on visual-mental representations than verbal ones.
The findings showed that in comparison to their general education peers, unaccommodated autistic students scored higher and solved math problems involving the identification of figures more quickly. Unaccommodated autistic students were also faster than their general education peers at solving the following types of math items: comparing measures using unit conversions, mentally rotating a triangle, interpreting linear equations, and constructing data analysis plots. Although autistic students who used the extended-time accommodation were lower performing than the other two groups, they had a higher accuracy rate on items involving identifying figures and calculating the diameter of a circle. Both groups of autistic students seem to perform poorer on word problems. Researchers concluded that the linguistic complexity could be one of the reasons that autistic students struggle with math word problems; however, there were two word problems with which they seemed to struggle despite the fact that they were not linguistically complex. It turns out that the items were rated as having substantial social context complexity. The researchers also looked at the student survey data on what types of math they enjoyed more and found they had more enjoyment working with shapes and figures and less enjoyment for solving equations.
The researchers recommend incorporating meta-cognitive and explicit schema instruction during mathematics instruction to aid autistic students in understanding real-life math word problems. They also recommend that assessment developers consider simplifying the language and social context of math word problems to make the assessment more equitable, fair, and accessible for autistic students. Because the autistic student population is particularly heterogenous, more research is required to better understand how to improve instructional strategies for them.
IES plans to release the same type of process data from the 2017 Grade 4 NAEP Mathematics at the end of this summer. We encourage researchers to request these process data to conduct research to understand test-taking behavior and performance of students with disabilities at the elementary school level. For a source of funding for the work, consider applying to the current Special Education Research Grants competition. Here are some important resources to support your proposal writing:
This blog was authored by Sarah Brasiel (Sarah.Brasiel@ed.gov), program officer at NCSER, and Juliette Gudknecht, summer data science intern at IES and graduate student at Teachers College, Columbia University. IES encourages special education researchers to use NAEP response process data for research under the Exploration project type within our standard Special Education Research Grants Program funding opportunity.