Project Activities
Through a series of cognitive interview studies, the research team will first examine how students respond to different feedback-and-revision formats using a variety of item types and item formats. Think-aloud protocols will be used to assess the frustration and motivation levels of students with and without disabilities under different feedback-and-revision, item type, and item format conditions. The feedback-and-revision format found to be most helpful to students with disabilities will be included in a large experimental design study examining the relationships between the provision of feedback-and-revision and the psychometric comparability and accuracy of the assessment scores.
Structured Abstract
Setting
The research will take place in school districts in Maryland and Georgia.
Sample
The target population consists of eighth grade special education students who have persistent learning problems in mathematics. During the development stage of the feedback and revision methodology, the researchers will sample 70 students with disabilities (target sample) and 45 students without disabilities (general education sample). For the experimental study, 200 students with disabilities who are eligible for alternate assessment (target sample) and 200 students without disabilities will participate in the study (general education sample).
Due to the nature of the research design, there is no intervention.
Research design and methods
Six cognitive interview studies will be conducted to determine how the target sample and the general education sample respond to two types of feedback-and-revision formats (answer-until-correct and second-chance), two different mathematics item types (numeric operation and geometry), and two different item formats (multiple-choice and constructed response). An experimental design study will follow to examine the impact of using a particular feedback-and-revision format on the psychometric comparability and accuracy of the assessment scores. The research team will examine whether providing this particular feedback-and-revision format to the target sample will improve their test performance, decrease their test anxiety, and increase their motivation.
Control condition
Due to the nature of the research design, there is no control condition.
Key measures
This project will use the following instruments: (1) Grade 8 National Assessment of Education Progress (NAEP) mathematics released items, (2) state assessment items, (3) students performance on state assessments, (4) researcher-developed cognitive interviews protocols, (5) State-Trait Anxiety Inventory, and (6) student and teacher surveys.
Data analytic strategy
The researchers will summarize the student responses collected in the cognitive interviews. The researchers will use classical test theory and item response theory to analyze student response data and to evaluate the reliability and validity of the assessment scores under the feedback-and-revision condition. The researchers will use multivariate analysis of variance to examine whether students' test performance, test anxiety levels, and motivation levels are different under the no feedback-and-revision and feedback-and-revision condition for the target and the general education sample.
People and institutions involved
IES program contact(s)
Products and publications
Products: The products of this research project include technical information on the reliability and validity of the feedback and revision method, presentations, and published reports.
ERIC Citations: Find available citations in ERIC for this award here.
Journal article, monograph, or newsletter
Attali, Y., Laitusis, C., and Stone, E. (2015). Differences in Reaction to Immediate Feedback and Opportunity to Revise Answers for Multiple-Choice and Open-Ended Questions. Educational and Psychological Measurement, 76(5): 787-802. doi:10.1177/0013164415612548 Full text
Nongovernment report, issue brief, or practice guide
Johnstone, C., Figueroa, C., Attali, Y., Stone, E., and Laitusis, C. (2013). Results of a Cognitive Interview Study of Immediate Feedback and Revision Opportunities for Students With Disabilities in Large-Scale Assessments (Synthesis Report 92). Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes. Full text
Questions about this project?
To answer additional questions about this project or provide feedback, please contact the program officer.
