Project Activities
The project team will conduct and disseminate secondary analyses of extant data from the 2017 NAEP 4th and 8th grade math tests as well as student, teacher, and school surveys to (a) determine which student-level variables are associated with eligibility for extended time on a test and actual use of extended time; (b) understand whether there are item characteristics (such as reading demands or location within the test) that predict item-level time use among students with disabilities eligible for extended time; and (c) compare selected time-use behaviors between students with disabilities eligible for extended time and students without disabilities to determine whether certain time-use sequences are differentially associated with item performance.
Structured Abstract
Setting
The data, which will be derived from computer-based administration of approximately 15-item math blocks from the 2017 NAEP mathematics assessment, was administered to a national sample of U.S. students.
Sample
The primary target sample will consist of 4th and 8th grade students with disabilities, within a nationally representative sample, who were eligible for extended time—based on their individualized education plan (IEP)—and participated in the corresponding computer-administered NAEP math test item blocks for which process data are available. The sample includes students across a variety of educational disability categories, with sufficient sample sizes to explore the data by disability for several high frequency categories (learning disability, emotional disturbance, speech/language impairment, and other health impaired). For several comparative analyses, researchers will also conduct analyses using random samples of students without disabilities from each grade who completed the corresponding test blocks.
Factors
The research team will explore a variety of student characteristics (such as demographics, math motivation, and achievement level) and item characteristics (such as reading demands, difficulty level, and location on test) as predictors of time use on items. Researchers will also explore specific item-level time-use behaviors (such as time spent using accessibility tools and time spent checking one's work) and actions (such as item revisits and rapid guessing), representing both regulated and dysregulated behaviors, and their association with item-level performance. These specific time-use behaviors are anticipated to be alterable via test signal innovations, interventions, and improvements in state testing policies and corresponding IEP decision-making.
Research design and methods
The research team will conduct secondary analysis of the 2017 NAEP mathematics assessment process data of 4th and 8th grade students. Researchers will use both descriptive and comparative quantitative analyses and data-mining approaches (for action sequences) to explore item-level and test-level student time-use behaviors, associated student- and item-level predictors, and how item-level time-use behaviors relate to item-level performance for students with disabilities eligible for extended time and a random sample of students without disabilities.
Control condition
Due to the nature of the research design, there is no control condition.
Key measures
Researchers will use data from the following sources: NAEP questionnaire data on individual students with disabilities completed by a school staff member familiar with the student's IEP, a student questionnaire to measure motivation in math and the test, raw test process data (time-stamped information about the student's specific keystrokes), math test block item content, and a features dataset (such as item response duration and item performance, already synthesized through the National Center for Education Statistics). Many of the variables are directly available within these datasets; others will be transformed using a priori definitions for time-use behaviors examined in prior published work or empirically derived using data-mining techniques.
Data analytic strategy
The researchers will use descriptive statistics, regression models, and process data visualization to address the research questions about item-level and test-level time-use and the extent to which various student- and item-level characteristics predict associated time-use behaviors and correct item responses. The researchers will also use a process-mining approach called N-grams to explore and compare the action sequences within test process data that correspond to correct versus incorrect responses to help identify time-use behaviors most likely to lead to correct responses.
Products and publications
Products: This project will result in targeted hypotheses about the patterns of time use that can be tested in future research and used for recommendations for improvements in accommodation decision-making. The research team will present findings in peer-reviewed journals and at conferences that target education researchers and test developers, practitioners (special educators and school psychologists), and professionals involved in state test policymaking. They will also provide a webinar to a variety of audiences at the end of the project.
ERIC Citations: Find available citations in ERIC for this award here.
Supplemental information
Co-Principal Investigators: Lovett, Benjamin J.; Buzick, Heather M.
Questions about this project?
To answer additional questions about this project or provide feedback, please contact the program officer.