Project Activities
Researchers will obtain restricted-use response process data along with student performance and contextual data from the 2017 NAEP Grade 8 mathematics assessment with a restricted use license. In this exploratory study, researchers will create novel and multidimensional measures of AF use and employ traditional analyses and emerging techniques in machine learning to explore AF utilization patterns. They will also employ quasi-experimental methods to examine the relationship between AF use and students' performance and test-taking behaviors. The findings will be disseminated to diverse key audiences, including teachers; district and school administrators; district, state, and federal policymakers; test developers; and researchers. The findings will improve understanding of AF use in digital assessments for SWD and provide data-based evidence for AF use for design-related guidelines and policies.
Structured Abstract
Sample
This study will use restricted-use response process data along with student performance and contextual data from the 2017 NAEP Grade 8 mathematics assessment. The overall national sample for the assessment comprised 148,100 students, but response process data is only available for one of the 10 blocks corresponding to about 28,000 students and one of 50 forms for about 2,800 students. The inclusion rate of SWD was 89%. SWDs represented 12% of all students who were assessed, and about 84% of SWDs were assessed with accommodations. Consequently, the results will be generalizable only to SWD who could take the NAEP assessment with accommodations, not the full population of SWD.
Research design and methods
The researchers will employ traditional analyses, emerging techniques in machine learning, and quasi-experimental designs to systematically explore the 2017 NAEP Grade 8 mathematics process data.
Key measures
To study AF availability and utilization, the research team will construct several measures using response process data. Measures of student demographics, teacher and school characteristics will come from the NAEP survey questionnaire. To evaluate student performance, they will use direct measures of performance including three outcomes that are available from NAEP response data: (1) mathematics performance on an individual item, (2) the number of correct items (across a block and a form); and (3) NAEP proficiency levels (basic, proficient, and advanced). They will also construct indirect measures of performance using response process data that focus on students' test-taking behaviors related to performance (such as the number of response changes). They will also use item characteristics defined by NAEP which include content areas (such as geometry), item difficulty, item complexity, item type (such as multiple-choice), and item sequence (order of presented items).
Data analytic strategy
Researchers will create novel and multidimensional measures of AF use. To explore AF utilization and utilization patterns, they will employ traditional analyses (e.g., descriptive statistics and regression analyses) and emerging techniques in machine learning and data mining (e.g., network analysis, latent class growth modeling, and process mining). They will also employ quasi-experimental methods (e.g., propensity score matching or Mahalanobis distance matching) to examine the relationship between AF use and students' performance and test taking behaviors.
People and institutions involved
IES program contact(s)
Products and publications
Products The products of this project will include academic journal articles, practitioner-oriented briefs, conference presentations (general research conferences, conferences focused on SWD, conferences on advanced analytics), social media posts, webinars or virtual online sessions, and publication venues allowing for interactive results using R.
Supplemental information
Co-Principal Investigator: Ruhan Circi, AIR
Questions about this project?
To answer additional questions about this project or provide feedback, please contact the program officer.
