Project Activities
The researchers selected and revised some narrative items from the existing MOCCA assessment and developed new narrative and expository items. All items were written or revised to be appropriate to themes that college students would encounter in their studies and experiences. Items and content were reviewed by an independent content-review panel. Through multiple iterations and pilot studies, the researchers used both classical test theory (CT) and item response theory (IRT) to refine these items based on item difficulty, item discrimination, and differential item functioning. Ultimately, they created three test forms of the assessment that were used in reliability and validity studies with a national sample of postsecondary students.
Structured Abstract
Setting
The early stages of the research was primarily conducted in 4-year postsecondary institutions (primarily urban settings) in California, Georgia, Minnesota, and North Dakota. The final stage of the assessment study involved a sample of primarily 4-year postsecondary students from across the United States.
Sample
Over 1100 postsecondary students participated in the initial development study, over 1600 postsecondary students participated in the first validation study, and 1614 students participated in the final national validation study. The mean age of participants in the final national validation study was 21.98 years (SD = 5.76). The sample identified as 64 percent female, 32 percent male, 3 percent genderfluid/nonconforming, and 1 percent did not respond. The vast majority (91 percent) were full-time students. The race/ethnicity of the sample was 47 percent White, 28 percent Latino/Hispanic, 10 percent Black/African American, 9 percent Asian, and 6 percent Other. A content-review panel consisted of six instructors and program directors who work with the target test population.
Assessment
The Multiple-choice Online Causal Comprehension Assessment-College (MOCCA-College) assesses postsecondary students' reading comprehension ability and distinguishes between common comprehension difficulties. MOCCA-College builds off an earlier measure, MOCCA. originally designed for third- to fifth-grade readers. The final MOCCA-C will consist of 3 scaled forms with 50 short narrative and informational texts that span grade 5 through 12 on the Flesch-Kincaid (FK) reading scale. Each seven-sentence story is missing the sixth sentence. Test takers choose a sentence from three options to fill the missing sentence. One of the options is correct and creates a causally coherent connection, one is a paraphrase, and one is an elaboration (i.e., an inference or association that is based on background knowledge, may be tangential, and isn't causally coherent). The final product also includes a mechanism to allow students to self-register and self-administer the assessment and automated and immediate feedback for test-takers with suggestions for improving monitoring and practicing inference generation while reading.
Research design and methods
The researchers selected a subset of original MOCCA items and wrote an additional 100 items. Postsecondary instructors, administrators, and experts reviewed those items, giving feedback on their content, readability, appropriateness, and cultural sensitivity. In phase 1, a sample of postsecondary students completed one of four forms of the MOCCA-College assessment, with a subset of students also completing think alouds. Using these data, the researchers refined the items and developed three forms of MOCCA-C. In phase 2, the research team began field testing the forms with an additional wave of postsecondary students and revised the forms or items if necessary. In the final year, the team conducted a national field test and evaluated the content, diagnostic, criterion, construct validity and fairness of the assessment.
Control condition
This research method does not use control conditions.
Key measures
Key measures included existing research-developed, standardized reading measures (namely the TOWRE-2 and the Nelson Denny Reading Test) and common postsecondary admissions data (such as ACT/ SAT scores and GPA).
Data analytic strategy
The researchers used item response theory (IRT), choice theory, logistic models, univariate and multivariate analyses, and reliability and validity analyses throughout all phases of the study.
Key outcomes
The main outcomes of the validation study are as follows:
People and institutions involved
IES program contact(s)
Products and publications
ERIC Citations: Find available citations in ERIC for this award here.
Project Website: https://mocca.uoregon.edu/#/
Additional Online Resources and Information:
- MOCCA-College (2021). EDGamesExpos 21. MOCCA-College by California State University, Chico. https://www.youtube.com/watch?v=bj-MkNU9kpg
- MOCCA-College. (2020). MOCCA-College Indeterminate Feedback https://www.youtube.com/watch?v=Fptj91qYmAE
- MOCCA-College. (2020). MOCCA-College Good Comprehender Feedback https://www.youtube.com/watch?v=_5YbQ4yojlI
- MOCCA-College. (2020). MOCCA-College Elaborator Feedback https://www.youtube.com/watch?v=7wen3G-5xg4
- MOCCA-College. (2020). MOCCA-College Paraphraser Feedback https://www.youtube.com/watch?v=BruOR9XZafA
Select Publications:
Journal articles
Inferential comprehension differences between narrative and expository texts: A systematic review and meta-analysisReading and Writing, 33,Full textClinton-Lisell, V., Taylor, T., Seipel, B., Carlson, S., & Davison, M. (2022). Performance on reading comprehension assessments and college achievement: A meta-analysis. Journal of College Reading and Learning, 52(3), 191-211. doi.org/10.1080/10790195.2022.2062626
Davison, M., Davenport, E. C., Jr., Hao, J., Seipel, B., Carlson, S. (2022). Regression with reduced rank predictor matrices: A model of trade-offs. Psychological Methods, 0(0). doi.org/10.1037/met0000512
Seipel, B., Kennedy, P., Carlson, S.E., Clinton-Lisell, V., & Davison, M. L. (2022) MOCCA-College: Preliminary validity evidence of a cognitive diagnostic reading comprehension assessment. Journal of Learning Disabilities, 0(0). Invited manuscript. doi.org/10.1177/00222194221121
Proceedings
Davison, M. L., Seipel, B., Clinton, V., Carlson, S., & Kennedy, P. (2020 ). MOCCA College: An assessment of inferential narrative and expository comprehension. Editorial Universitat Politècnica de València. 417-425.
Project website:
Related projects
Supplemental information
Co-Principal Investigators: Carlson, Sarah; Davison, Mark; Clinton-Lisell, Virginia
- The initial pilot study and subsequent field tests established great internal reliability of the assessment, both for in terms of comprehension skill and comprehender subtypes (Seipel et al., 2022).
- Both pilot study and field test indicate that expository and narrative comprehension (in the context of MOCCA-College) is a unidimensional construct although generally expository texts are more difficult to read than narrative texts (Davison et al., 2020; Clinton et al., 2020).
- Reading comprehension remains a predictor of college achievement and is correlated with other college admission assessments (Clinton-Lisell et al., 2022 )
Questions about this project?
To answer additional questions about this project or provide feedback, please contact the program officer.
