|Title:||Multiple-choice Online Causal Comprehension Assessment for Postsecondary Students (MOCCA-College): Measuring Individual Differences in Reading Comprehension Ability of Struggling College Readers by Text Type|
|Principal Investigator:||Seipel, Ben||Awardee:||California State University, Chico|
|Program:||Postsecondary and Adult Education [Program Details]|
|Award Period:||3 years (07/01/2018 - 06/30/2021)||Award Amount:||$1,383,123|
Co-Principal Investigators: Carlson, Sarah; Davison, Mark; Clinton-Lisell, Virginia
Purpose: The research team refined and validated a reading comprehension assessment (MOCCA), which was the product of an IES-funded project (R305A140185) and was designed for and validated with third- through fifth-grade students, for use with postsecondary students. The refined assessment, MOCCA-College (MOCCA-C) is a practical, reliable, and valid measure of reading comprehension processes that can help postsecondary practitioners, administrators, and researchers diagnose students' reading comprehension difficulties.
Project Activities: The researchers selected and revised some narrative items from the existing MOCCA assessment and developed new narrative and expository items. All items were written or revised to be appropriate to themes that college students would encounter in their studies and experiences. Items and content were reviewed by an independent content-review panel. Through multiple iterations and pilot studies, the researchers used both classical test theory (CT) and item response theory (IRT) to refine these items based on item difficulty, item discrimination, and differential item functioning. Ultimately, they created three test forms of the assessment that were used in reliability and validity studies with a national sample of postsecondary students.
Key Outcomes: The main outcomes of the validation study are as follows:
Setting: The early stages of the research was primarily conducted in 4-year postsecondary institutions (primarily urban settings) in California, Georgia, Minnesota, and North Dakota. The final stage of the assessment study involved a sample of primarily 4-year postsecondary students from across the United States.
Sample: Over 1100 postsecondary students participated in the initial development study, over 1600 postsecondary students participated in the first validation study, and 1614 students participated in the final national validation study. The mean age of participants in the final national validation study was 21.98 years (SD = 5.76). The sample identified as 64 percent female, 32 percent male, 3 percent genderfluid/nonconforming, and 1 percent did not respond. The vast majority (91 percent) were full-time students. The race/ethnicity of the sample was 47 percent White, 28 percent Latino/Hispanic, 10 percent Black/African American, 9 percent Asian, and 6 percent Other. A content-review panel consisted of six instructors and program directors who work with the target test population.
Assessment: The Multiple-choice Online Causal Comprehension Assessment-College (MOCCA-College) assesses postsecondary students' reading comprehension ability and distinguishes between common comprehension difficulties. MOCCA-College builds off an earlier measure, MOCCA. originally designed for third- to fifth-grade readers. The final MOCCA-C will consist of 3 scaled forms with 50 short narrative and informational texts that span grade 5 through 12 on the Flesch-Kincaid (FK) reading scale. Each seven-sentence story is missing the sixth sentence. Test takers choose a sentence from three options to fill the missing sentence. One of the options is correct and creates a causally coherent connection, one is a paraphrase, and one is an elaboration (i.e., an inference or association that is based on background knowledge, may be tangential, and isn't causally coherent). The final product also includes a mechanism to allow students to self-register and self-administer the assessment and automated and immediate feedback for test-takers with suggestions for improving monitoring and practicing inference generation while reading.
Research Design and Methods: The researchers selected a subset of original MOCCA items and wrote an additional 100 items. Postsecondary instructors, administrators, and experts reviewed those items, giving feedback on their content, readability, appropriateness, and cultural sensitivity. In phase 1, a sample of postsecondary students completed one of four forms of the MOCCA-College assessment, with a subset of students also completing think alouds. Using these data, the researchers refined the items and developed three forms of MOCCA-C. In phase 2, the research team began field testing the forms with an additional wave of postsecondary students and revised the forms or items if necessary. In the final year, the team conducted a national field test and evaluated the content, diagnostic, criterion, construct validity and fairness of the assessment.
Key Measures: Key measures included existing research-developed, standardized reading measures (namely the TOWRE-2 and the Nelson Denny Reading Test) and common postsecondary admissions data (such as ACT/ SAT scores and GPA).
Data Analytic Strategy: The researchers used item response theory (IRT), choice theory, logistic models, univariate and multivariate analyses, and reliability and validity analyses throughout all phases of the study.
Related IES Projects: Multiple-choice Online Cloze Comprehension Assessment (MOCCA): Refining and Validating a Measure of Individual Differences in Reading Comprehension Processes During Reading (R305A140185), Multiple-choice Online Causal Comprehension Assessment Refinement: Achieving Better Discrimination via Experimental Item Types and Adaptive Testing (R305A190393)
ERIC Citations: Find available citations in ERIC for this award here.
Project Website: https://mocca.uoregon.edu/#/
Additional Online Resources and Information:
Clinton, V., Taylor, T., Bajpayee, S., Seipel, B., Carlson, S.E., & Davison, M. (2020). Inferential comprehension differences between narrative and expository texts: A systematic review and meta-analysis. Reading and Writing, 33, 2223-2248. doi: 10.1007/s11145-020-10044-2 Full text
Clinton-Lisell, V., Taylor, T., Seipel, B., Carlson, S., & Davison, M. (2022). Performance on reading comprehension assessments and college achievement: A meta-analysis. Journal of College Reading and Learning, 52(3), 191–211. doi.org/10.1080/10790195.2022.2062626
Davison, M., Davenport, E. C., Jr., Hao, J., Seipel, B., Carlson, S. (2022). Regression with reduced rank predictor matrices: A model of trade-offs. Psychological Methods, 0(0). doi.org/10.1037/met0000512
Seipel, B., Kennedy, P., Carlson, S.E., Clinton-Lisell, V., & Davison, M. L. (2022) MOCCA-College: Preliminary validity evidence of a cognitive diagnostic reading comprehension assessment. Journal of Learning Disabilities, 0(0). Invited manuscript. doi.org/10.1177/00222194221121
Carlson, S. E., Seipel, B., Biancarosa, G., Davison, M. L., & Clinton, V. (2019). Demonstration of an innovative reading comprehension diagnostic tool. In M. Scheffel, J. Broisin, V. Pammer-Schindler, A. Ioannou, & J. Schneider (Eds.), Transforming Learning with Meaningful Technologies, (pp.769–772), Springer Nature Switzerland AG. doi.org/10.1007/978-3-030-29736-7_85
Davison, M. L., Seipel, B., Clinton, V., Carlson, S., & Kennedy, P. (2020). MOCCA College: An assessment of inferential narrative and expository comprehension. Editorial Universitat Politècnica de València. 417–425.