Co-Principal Investigator: Ruhan Circi, AIR

Purpose Current knowledge about the use of accessibility features (AFs)—consisting of accommodations available only to students with disabilities (SWD) and universal design (UD) elements available to all students—is largely limited to information on students' interaction with mostly paper-and-pencil tests. Little is known about whether SWDs use the AFs available to them; what features or combination of features they use; the extent of use if they do use the features; and whether certain item, student, and school characteristics elicit use of AFs. The assumption that accommodations level the playing field (i.e., improve the testing performance of SWD) has not been tested in digitally based assessments (DBAs). This project seeks to systematically explore the 2017 NAEP Grade 8 mathematics process data to provide empirical evidence on the utilization of AFs, including accommodations and UD elements; how item characteristics, student characteristics, and school characteristics are related to the utilization of AFs; and how AF utilization relates to students' test-taking behavior and performance. The results from this study will help improve the validity of results from such DBAs for SWD and inform ongoing test development, with a focus on minimizing the impact of construct-irrelevant factors on student achievement in a DBA. The results of this study can be translated into actionable policies (e.g., decisions on use of UD elements), guidelines (e.g., conditions under which accessibility features may benefit students), and tools for school administrators and teachers.

Project Activities Researchers will obtain restricted-use response process data along with student performance and contextual data from the 2017 NAEP Grade 8 mathematics assessment with a restricted use license. In this exploratory study, researchers will create novel and multidimensional measures of AF use and employ traditional analyses and emerging techniques in machine learning to explore AF utilization patterns. They will also employ quasi-experimental methods to examine the relationship between AF use and students' performance and test-taking behaviors. The findings will be disseminated to diverse key audiences, including teachers; district and school administrators; district, state, and federal policymakers; test developers; and researchers. The findings will improve understanding of AF use in digital assessments for SWD and provide data-based evidence for AF use for design-related guidelines and policies.

Products The products of this project will include academic journal articles, practitioner-oriented briefs, conference presentations (general research conferences, conferences focused on SWD, conferences on advanced analytics), social media posts, webinars or virtual online sessions, and publication venues allowing for interactive results using R.

Structured Abstract

Sample This study will use restricted-use response process data along with student performance and contextual data from the 2017 NAEP Grade 8 mathematics assessment. The overall national sample for the assessment comprised 148,100 students, but response process data is only available for one of the 10 blocks corresponding to about 28,000 students and one of 50 forms for about 2,800 students. The inclusion rate of SWD was 89%. SWDs represented 12% of all students who were assessed, and about 84% of SWDs were assessed with accommodations. Consequently, the results will be generalizable only to SWD who could take the NAEP assessment with accommodations, not the full population of SWD.

Research Design and Methods The researchers will employ traditional analyses, emerging techniques in machine learning, and quasi-experimental designs to systematically explore the 2017 NAEP Grade 8 mathematics process data.

Key Measures To study AF availability and utilization, the research team will construct several measures using response process data. Measures of student demographics, teacher and school characteristics will come from the NAEP survey questionnaire. To evaluate student performance, they will use direct measures of performance including three outcomes that are available from NAEP response data: (1) mathematics performance on an individual item, (2) the number of correct items (across a block and a form); and (3) NAEP proficiency levels (basic, proficient, and advanced). They will also construct indirect measures of performance using response process data that focus on students' test-taking behaviors related to performance (such as the number of response changes). They will also use item characteristics defined by NAEP which include content areas (such as geometry), item difficulty, item complexity, item type (such as multiple-choice), and item sequence (order of presented items).

Data Analytic Strategy Researchers will create novel and multidimensional measures of AF use. To explore AF utilization and utilization patterns, they will employ traditional analyses (e.g., descriptive statistics and regression analyses) and emerging techniques in machine learning and data mining (e.g., network analysis, latent class growth modeling, and process mining). They will also employ quasi-experimental methods (e.g., propensity score matching or Mahalanobis distance matching) to examine the relationship between AF use and students' performance and test taking behaviors.