|Title:||Multiple-choice Online Cloze Comprehension Assessment (MOCCA): Refining and Validating a Measure of Individual Differences in Reading Comprehension Processes During Reading|
|Principal Investigator:||Biancarosa, Gina||Awardee:||University of Oregon|
|Program:||Reading and Writing [Program Details]|
|Award Period:||3 years (7/1/2014–6/30/2017)||Award Amount:||$1,599,776|
Co-Principal Investigator: Sarah E. Carlson (University of Oregon), Ben Seipel (California State University, Chico), Mark Davison (University of Minnesota)
Purpose: Numerous children struggle with reading comprehension and previous research shows that a subset of these children struggle with higher-level reading skills, such as maintaining causal coherence while reading. Causally coherent inferences require the reader to synthesize why an event occurs based on relevant goals previously identified in the text and to generate missing information from background knowledge. Students who struggle with causal coherence typically fall into one of two types and tend to respond differently to intervention depending on their type: the first type tends to paraphrase what they have already read; the second type tends to make lateral connections, or inferences or personal associations which are not causally related to the text. However, without an assessment to identify the type of poor comprehender, teachers may be unable to provide the appropriate intervention for each student. The Multiple-choice Online Cloze Comprehension Assessment (MOCCA) is a paper-and-pencil assessment to diagnose specific types of poor comprehension in third through fifth grade students. Each multiple-choice question has a correct answer, an answer that indicates the student has paraphrased, and an answer that indicates a lateral connection. The current project will refine, expand, and validate the MOCCA to diagnose the type of poor comprehender for students in grades three through five, including creating multiple forms for each grade level and converting the test to an online platform.
Project Activities: The goal of this project is to refine, expand, and validate the MOCCA. The project team will develop an initial pool of 150 items for each grade. Teachers will provide content validity for the items. Items will be pilot tested in the first year and field tested in two large-scale tests in Years 2 and 3. Additionally, the team will work on moving the administration of MOCCA from paper-and-pencil to a computer-based assessment. Third- through fifth-grade students will complete think alouds in each year in order to provide information to the researchers regarding how the items are functioning.
Products: The final product of this study will be the fully developed and validated Multiple-choice Online Cloze Comprehension Assessment (MOCCA) for students in third through fifth grades. Products will also include peer reviewed publications.
Setting: This study takes place in school districts in Oregon and California. Additionally, a nationally-representative sample of students from around the U.S. will participate in the field trials.
Sample: Participants include approximately 600 third through fifth grade students from California, 600 third through fifth grade students from Oregon, and 9,000 third through fifth grade students from other districts around the country.
Intervention: Researchers plan to refine and validate the Multiple-choice Online Cloze Comprehension Assessment (MOCCA), an assessment designed to diagnose specific types of poor comprehension in third- through fifth-grade students. The final product will be computer based and will include three forms per grade. Each item on the assessment will be a seven-sentence narrative with the sixth sentence deleted from the narrative. The missing sixth sentence resolves the goal of the story. Students will be asked to choose the best sentence from among four choices to insert as the sixth sentence. One of the answers will be correct (i.e. is causally coherent with the narrative), one of the answers will be a paraphrase, and one of the answers will be a lateral connection (i.e. an inference or personal association which is not causally coherent with the text). The research team anticipates each form including approximately 40 items.
Research Design and Methods: In order to create additional forms of the assessment for each grade level, the research team will begin by developing an initial pool of 150 items for each grade level. Teachers will provide content validity feedback. In Year 1, the MOCCA items will be piloted with students in third through fifth grade. A subsample of students will also complete think alouds to provide information regarding the functioning of the items. In Year 2, following revisions to the items, students will participate in the first MOCCA field test. Again, think aloud data will be collected from a subsample of students. Following more revisions, a second field test, with accompanying think alouds, will be conducted in Year 3. Content, diagnostic, criterion, and construct validity will be assessed, as well as a fairness assessment.
Control Condition: Due to the nature of this research design, there is no control condition.
Key Measures: Criterion validity will be established by correlating the MOCCA with school-administered measures of reading including easyCBM Vocabulary, the Dynamic Indicators of Basic Early Literacy Skills (DIBELS), and the state’s standardized reading test (e.g. SmarterBalanced or Standardized Testing and Reporting (STAR) in California). Construct validity will be established by comparing the MOCCA with the Bridge-IT Picture Version and the Diagnostic Assessment of Reading Comprehension (DARC).
Data Analytic Strategy: During the refinement stage of the project, researchers will use data from the think alouds to inform revisions of the incorrect answer choices and to identify any additional profiles of poor comprehenders. Data will be analyzed using two Item Response Theory (IRT) analyses. The first analysis will be a multidimensional IRT based on a full decision choice model which produces three scores for each person: a causal coherence score, a paraphrase score, and a lateral connection score. The second analysis will be a unidimensional 2PL model. Correlations will be used to establish validity.
Journal article, monograph, or newsletter
Davison, M.L., Biancarosa, G., Carlson, S.E., Seipel, B., and Liu, B. (2017). Preliminary Findings on the Computer-Administered Multiple-Choice Online Causal Comprehension Assessment, A Diagnostic Reading Comprehension Test. Assessment for Effective Intervention.