Previous Award Number: R324P210005
Previous Awardee: SRI International

Co-Principal Investigator: Susu Zhang, University of Illinois, Urbana-Champaign; Jennifer Yu, SRI International

Purpose This study will use NAEP grade 8 mathematics assessment process data to understand the action and time sequences of key features in test-taking behavior for learners with and without disabilities, and how the test-taking behavior reflected through these key features relates to student outcomes. Although any performance or process differences found between learners may be due to true differences in test takers' mathematics problem solving, it is possible that the content and design of certain items make them harder for learners with disabilities. Thus, the proposed analysis of NAEP process data has two ultimate goals: (1) to provide insights on how educators can effectively teach learners with disabilities to use successful strategies to solve math problems, and (2) to reveal methods to design and evaluate equitable and accurate assessments, thus improving item fairness and accommodation features to increase engagement and accessibility by learners with different abilities.

Project Activities The recent shift to NAEP digital assessments affords researchers with exciting opportunities to delve into test takers' observable actions and behaviors, which are stored in computer-generated log files as sequences of events with time stamps. The 2017 NAEP assessments collected data on the outcome of a task and on actions that reflect cognitive processes occurring during task completion. Using these data as well as student, teacher, and school surveys, researchers will conduct several activities:

• Use state-of-the-art machine learning techniques to extract action features and time features from NAEP grade 8 math assessment process data, which reflect four key components of learners' test-taking behaviors: underlying math cognitive processes, time on task, levels of engagement, and use of accommodations and accessibility supports.
• Examine how these test-taking behavior features differ by disability status.
• Identify patterns of test-taking behavior that lead to successful performance on NAEP mathematics assessment items.
• Investigate differences in the association between test-taking behaviors and test performance for learners with disabilities compared to their peers without disabilities.
• Use a structural equation modeling (SEM) framework to understand whether the interrelationships among instruction, math performance, and the four components of testing behavior differ by disability status.

Products The project will share interim findings about machine learning techniques to extract action features and time features from NAEP process data and challenges and benefits of the application of these methods. The project will also result in peer-reviewed publications and presentations as well as additional dissemination products that reach education stakeholders such as practitioners and policymakers.

Structured Abstract

Sample In 2017, the NAEP mathematics assessment sampled 144,900 grade 8 students from 6,500 schools—all of whom took the assessment on digital tablets. Our analytic sample will include data from approximately 28,000 students. Comprised of the NAEP restricted-use response process data and associated files, this includes data from those who took the digitally designed released block and those who took both blocks of released cognitive items (approximately 2,800 students).

Research Design and Methods Researchers will use machine-learning techniques to extract action features and time features from NAEP process data that are hypothesized to reflect four key components of learners' test-taking behavior in mathematics: underlying math cognitive process, time on task, levels of engagement, and use of accommodations and accessibility supports. Researchers then will compare these test-taking features between learners with disabilities and their peers without disabilities. Next, researchers will use predictive models to estimate outcome data from extracted testing-behavior features for learners with and without disabilities. Lastly, researchers will integrate survey data into the analysis using a SEM framework to understand the interrelationships among math instruction, performance, and testing behaviors, and how these interrelationships differ by disability status.

Key Measures First, this study will extract key action and time features from the NAEP process data which include but are not limited to the following: process length (i.e., number of actions taken), number of revisits the test takers make to the same item, total response time, time spent on key actions, number of subtasks completed out of the total subtasks, number of times clicked a certain button (e.g., clear answer), and action sequence and time sequence of use of tools (i.e., calculator, zoom, and other). Unsupervised machine learning techniques will extract additional features that can reveal objective information about mathematics test-taking behavior that is not identified (or overlooked) by expert opinion. Other measures include math outcomes (raw response, scored response, and percent correct), student demographics, and administrator, teacher, and student survey data.

Data Analytic Strategy The researchers will use a variety of machine-learning strategies to extract math performance and behavior features from the process data including both (1) unsupervised learning techniques (n-grams, multidimensional scaling, and sequence-to-sequence autoencoders), and (2) supervised learning techniques (classification and regression tree, gradient boosting, random forest, and support vector machine). The project will use subject matter experts to interpret the principal clusters/features and assign them meaning based on the proposed dimensions of math test-taking behavior. Finally, SEM will be used to integrate the feature data with findings from survey data to determine interrelationships among student background, item characteristics, and teacher instructional practices. This analysis will improve the classification system and model prediction of performance outcomes from disability status and item characteristics.