Project Activities
The research team developed two forms of the Inference Making and Reasoning in Biology (IMRB) measure. Existing items (form A) were IRT scaled and cognitive interviews were collected from students who had completed a biology course. New texts were chosen, a different sample of biology students completed think-aloud protocols and correct and incorrect deductions (bridging inferences) were identified from those. A second set of questions (form B) was written, cognitive interviews were collected from those, and IRT scaling was applied to form B. The researchers also wrote a comprehensive technical manual including directions for end users. Data from one university and the manual were submitted to the ICPSR repository (deposit #TEST-120256).
Structured Abstract
Setting
This project took place at postsecondary institutions in Pennsylvania and Illinois.
Sample
Participants included undergraduate students enrolled in or who had just completed introductory biology courses. These 2,551 students participated in 5 studies at 2 universities over 3 years. Of these, 211 provided think-aloud or cognitive interview data used to develop and validate the measure. The remaining 2,340 provided multiple-choice response data, used for reliability, validity, and fairness (DIF) analyses.
Research design and methods
This project used an iterative measure development and refinement process to create two equivalent versions of our biology reasoning measure. In this process, the team audiotaped students' thinking about biology texts, and identified correct and incorrect conclusions drawn from the biology material. Texts were then shortened, questions were written tapping onto the correct and incorrect reasoning shown by students, and multiple-choice answers were written. The draft measures were given to biology students, who also verbalized their thinking while answering the reasoning questions. These interviews showed that, indeed, the measure requires deductive reasoning but notprior knowledge, test-taking strategies, other learning strategies such as summarizing, or vocabulary. Each version of the multiple-choice measure was then given to large samples of biology students. Analyses suggested a few of the newly developed questions should be dropped; they also showed that both forms met a very high standard for quality measures and could be used to predict student course grades.
Key measures
In addition to the two forms, the research team collected data on students' background and course grades.
Data analytic strategy
Researchers used the 3PLM model for IRT scaling, and the Mantel-Haentzel chi squared test for detecting DIF. They used ESEM with earlier samples and CFA with later samples to provide evidence for unidimensionality. Think-aloud protocols were analyzed using path models, and cognitive interviews were analyzed using non-parametric dependent-samples t tests.
Key outcomes
The main findings of the project are as follows:
People and institutions involved
IES program contact(s)
Products and publications
ERIC Citations: Find available citations in ERIC for this award here.
Publicly available data: Cromley, Jennifer. Inference-Making and Reasoning: Refinement of an Assessment for Use in Gateway Biology Courses, Illinois, 2018-2019. ICPSR38276-v1. Ann Arbor, MI: Inter-university Consortium for Political and Social Research [distributor], 2021-12-07: https://www.icpsr.umich.edu/web/ICPSR/studies/38276/versions/V1. Data are restricted for research use, and requestors will have to comply with ICPSR restrictions to access the files.
Project website: https://www.cromleylab.org/copy-of-meta-analysis-of-multimedia
Select Publications
Journal Articles
Cromley, J. G., Dai, T., Fechter, T. S., Nelson, F. E., Van Boekel, M., & Du, Y. (2021). Development of a tool to assess inference-making and reasoning in biology. Journal of Microbiology and Biology Education, 22(2), e00159-21. https://doi.org/10.1128/jmbe.00159-21
Cromley, J. G., Dai, T., Fechter, T., Van Boekel, M., Nelson, F. E., and Dane, A. (2021). What cognitive interviewing reveals about a new measure of undergraduate biology reasoning. The Journal of Experimental Education, 89(1), 145-168. Full Text
Cromley, J. G., Ma, S., Van Boekel, M., & Dane, N. (2020). Pickup of causal language and inference during and after reading illustrated text. Reading Psychology, 41(3), 157-182. doi.org/10.1080/02702711.2020.1768974 Dai, T., Van Boekel, M., Cromley, J. Nelson, F. and Fechter, T. (2018). Using think-alouds to create a better measure of biology reasoning. SAGE Research Methods Cases [online]. doi: 10.4135/9781526437167 Full Text
Supplemental information
Co-Principal Investigators: Dai, Ting; Fechter, Tai (Sukin); Nelson, Frank E.
- Both forms of the IMRB are reliable, valid, and fair across sex, race, and SES groups of introductory undergraduate biology students (Cromley, Dai, Fechter, Van Boekel, Nelson, & Dane, 2021).
- The measure explains significant variance in course grades, either in combination with SAT/ACT scores or by itself (e.g., if a university has dropped its standardized test requirement)(Cromley, Dai, Fechter, Nelson, Van Boekel, & Du, 2021).
- Students who learn biology and take a 'facts-first' approach such as listing many nouns but not connecting them with verbs, have particularly low learning outcomes (Cromley, Ma, Van Boekel, & Dane, 2020).
Questions about this project?
To answer additional questions about this project or provide feedback, please contact the program officer.
