Project Activities
Using restricted-use response process data along with student performance and contextual data from the 2017 NAEP Grade 8 mathematics assessment, the researchers created novel and multidimensional measures of accessibility features use and employed traditional analyses and emerging techniques in machine learning to explore accessibility features utilization patterns. They also employed quasi-experimental methods to examine the relationship between accessibility features use and students' performance and test-taking behaviors.
Structured Abstract
Sample
This study used restricted-use response process data along with student performance and contextual data from the 2017 NAEP Grade 8 mathematics assessment. The overall national sample for the assessment comprised 148,100 students, but response process data was only available for one of the 10 blocks corresponding to about 28,000 students and one of 50 forms for about 2,800 students. The inclusion rate of SWD was 89%. SWDs represented 12% of all students who were assessed, and about 84% of SWDs were assessed with accommodations. Consequently, the results will be generalizable only to SWD who could take the NAEP assessment with accommodations, not the full population of SWD.
Research design and methods
The researchers employed traditional analyses, emerging techniques in machine learning, and quasi-experimental designs to systematically explore the 2017 NAEP Grade 8 mathematics process data.
Key measures
To study accessibility features availability and utilization, the research team constructed several measures using response process data. Measures of student demographics and teacher and school characteristics came from the NAEP survey questionnaire. To evaluate student performance, they used direct measures of performance including three outcomes that are available from NAEP response data: (1) mathematics performance on an individual item, (2) the number of correct items (across a block and a form); and (3) NAEP proficiency levels (basic, proficient, and advanced). They also constructed indirect measures of performance using response process data that focus on students' test-taking behaviors related to performance (such as the number of response changes). They also used item characteristics defined by NAEP, which include content areas (such as geometry), item difficulty, item complexity, item type (such as multiple-choice), and item sequence (order of presented items).
Data analytic strategy
Researchers created novel and multidimensional measures of accessibility features use. To explore accessibility features utilization and utilization patterns, they employed traditional analyses (e.g., descriptive statistics and regression analyses) and emerging techniques in machine learning and data mining (e.g., network analysis, latent class growth modeling, and process mining). They also employed quasi-experimental methods (e.g., propensity score matching or Mahalanobis distance matching) to examine the relationship between accessibility features use and students' performance and test taking behaviors.
Key outcomes
The main findings of this project, as reported by the principal investigator, are as follows:
- Students do not always use the accessibility features available to them. Approximately half of the students utilized scratchwork, a third used extended time, and less than 25% used equation editors and text-to-speech. SWD were more likely to use the equation editor and text-to-speech than their peers without disabilities.
- Not all accessibility feature use leads to better performance. Extended time usage significantly improves test performance for students with disabilities. While scratchwork is associated with improved test performance for SWD, the association between both text-to-speech and equation editor and student performance is mixed, suggesting potential distractions rather than support for some students.
- More students than allowed may benefit from extended time. The majority of the students who qualified for extended time accommodations did not utilize it whereas about a quarter of the students without extended time accommodations received a time out message while they worked past the allotted time. Our findings show that early testing behaviors could potentially identify students who might benefit from extended time accommodations.
- Students who properly use the accessibility features in the tutorial tend to experience better outcomes in the subsequent assessment.
- Increased reading difficulty is associated with a higher likelihood of using text-to-speech and zoom functions, but a lower likelihood of using the highlight function. SWD exhibit a higher likelihood of using the highlight function and both a higher likelihood and frequency of using the zoom function compared to their peers without disabilities when faced with increased reading difficulty.
- Process mining analyses, which use the sequences of student activities, are useful in identifying problem-solving strategies and misconceptions. These analyses have revealed distinct approaches employed by students when solving algebra problems and have identified unique patterns, particularly among SWD.
- Studying revisit behaviors (total time, average time, and type of revisit) helps identify student item revisit patterns. SWD are more likely to skip or reattempt responses during revisits. Analyses identified four distinct groups of students based on revisit behaviors: those who spend more time on later revisits, those with short revisit times, those who spend more time on earlier revisits, and those with mixed revisit times.
People and institutions involved
IES program contact(s)
Project contributors
Products and publications
Publications:
ERIC Citations: Find available citations in ERIC for this award here.
Select Publications:
Ogut, B., Webb, B., Hicks, J., Circi, R., & Yin, M. (2024). Exploring Mathematical Problem-Solving Through Process Mining: Insights from Large-Scale Assessment Log Data. Computers in the Schools, 1–31. https://doi.org/10.1080/07380569.2024.2416422
Supplemental information
Co-Principal Investigator: Ruhan Circi, AIR
Questions about this project?
To answer additional questions about this project or provide feedback, please contact the program officer.
