|Multiple-choice Online Causal Comprehension Assessment Refinement: Achieving Better Discrimination via Experimental Item Types and Adaptive Testing
|University of Oregon
|Literacy [Program Details]
|3 Years (09/01/2019 - 08/31/2022)
Co-Principal Investigator: Davison, Mark L.; Kennedy, Patrick C.; Weiss, David J.
Purpose: This project will develop and refine a computer-adaptive version of MOCCA (Multiple-choice Online Cloze Comprehension Assessment), an existing reading comprehension measure for third- through fifth-grade readers. MOCCA helps distinguish between students who make one of two common comprehension errors. Students with weak comprehension skills tend either to paraphrase (rephrase prior information from the text but do not generate missing information) or to make lateral connections (make an elaborative inference or a personal association). Although the current version helps identify between these patterns of errors, MOCCA can be improved to better meet the needs of students and teachers. By expanding the item bank and making MOCCA computer-adaptive, MOCCA will help teachers gather both formative and summative information about students and reduce test-taking time and burden.
Project Activities: The work will occur in three work phases. First, the research team will create and validate additional assessment items with a greater range of difficulty and an expanded range of item formats. Concurrently, the researchers will develop a computer-adaptive testing (CAT) model for MOCCA. Finally, they will establish the technical adequacy of the CAT version of MOCCA for both screening and progress monitoring. These phases will leverage iterative rounds of simulation and pilot studies, consultations with research experts and teachers, and large-scale field tests with nationally representative samples.
Products: The final products will include a normed CAT version of MOCCA suitable for use in grades 3 through 5 that will take 15 to 20 minutes to administer and MOCCA user materials. The research team will also produce peer-reviewed publications.
Setting: The primary research sites for the development work are Oregon and Georgia. A nationwide sample will participate in the final validation.
Sample: The sample includes over 3000 public school students in grades 3 to 5 with an over-sampling of key minority populations based on race/ethnicity and special services status.
Assessment: The CAT version of MOCCA builds off the computer-based, MOCCA, a reading assessment for third- through fifth-grade students developed and validated under a previous IES-funded project (R305A140185). The original, computer-based MOCCA uses a seven-sentence narrative in which the sixth sentence is deleted. Students must select the missing sixth sentence from among three choices. One of the options is correct and creates a causally coherent connection, one is a paraphrase, and one is a lateral connection, namely an inference or association that is based on background knowledge, may be tangential, and is not causally coherent. The CAT version of MOCCA will continue to have the same general design but with a few changes. The next version of MOCCA will include items with more sentences overall or with five responses options (two of each incorrect type) for the upper grade levels in an attempt to increase the number of items with higher difficulties. It will be adaptive, thereby reducing the overall testing time. And it will have more items, thereby allowing teachers to use it throughout the school year and across years to track student progress and assess student achievement.
Research Design and Methods: The project has three aims: to develop more items and increase some items' difficulty, to build the CAT version, and to validate the CAT MOCCA version for formative and summative use. During year 1 and 2, the researchers will develop new items by collecting input and feedback from experts, revising the previous MOCCA's item writing specifications, and writing new items that are longer and have more incorrect responses. At the same time, they will program the CAT software and develop simulation software that will allow them to simulate students' progress through the CAT version. Near the end of year 2, they will run a series of mini pilots of the CAT MOCCA with new items to test the administration software with real students using the full set of potential items. They will also conduct a larger field test to improve the CAT administrations and to complete the scaling of the items. After finalizing the items and the CAT administration, they will conduct random-assignment large-scale studies of the CAT version during the typical benchmarking periods in schools (namely August-October, December-February, and April-June) to validate CAT MOCCA against the current MOCCA and other reading measures. They will also conduct two additional administrations using only the CAT MOCCA to explore MOCCA's sensitivity to change for the more frequent assessment administrations typically employed under progress-monitoring.
Control Condition: The research team will randomly assign students within both grade and classrooms, students will be randomly assigned to either the fixed MOCCA or CAT MOCCA.
Key Measures: Key measures include curriculum-based measures (CBMs) of reading and math and state test data such as DIBELS and easy CBM. They will also collect demographic data on gender, race/ethnicity, language status, special education status, and meals status from schools to determine whether there are systematic differences among groups.
Data Analytic Strategy: The researchers will use item response theory (IRT) to estimate item difficulties, and they will conduct differential item functioning (DIF) analyses and teacher content review panels will establish item fairness.
Related IES Projects: Multiple-choice Online Cloze Comprehension Assessment (MOCCA): Refining and Validating a Measure of Individual Differences in Reading Comprehension Processes During Reading (R305A140185); Multiple-choice Online Causal Comprehension Assessment for Postsecondary Students (MOCCA-College): Measuring Individual Differences in Reading Comprehension Ability of Struggling College Readers by Text Type (R305A180417)