Co-Principal Investigators: Dai, Ting; Fechter, Tai (Sukin); Nelson, Frank E.

Purpose: In this project, the researchers developed a measure of deductive reasoning for postsecondary introductory biology students presented with new information. First-year biology college courses constitute a gateway for success in multiple STEM majors, yet many academically able students struggle to learn the concepts. Previous research suggested that taking in large amounts of new information and being able to draw one's own conclusions with that information might be a malleable skill that contributes to course performance. To identify such a skill for biology instructors across diverse higher education settings, the researchers created two versions of a multiple-choice reasoning measure which are reliable, valid, fair, and predict achievement beyond SAT/ACT scores in first-year college biology courses.

Project Activities: The research team developed two forms of the Inference Making and Reasoning in Biology (IMRB) measure. Existing items (form A) were IRT scaled and cognitive interviews were collected from students who had completed a biology course. New texts were chosen, a different sample of biology students completed think-aloud protocols and correct and incorrect deductions (bridging inferences) were identified from those. A second set of questions (form B) was written, cognitive interviews were collected from those, and IRT scaling was applied to form B. The researchers also wrote a comprehensive technical manual including directions for end users. Data from one university and the manual were submitted to the ICPSR repository (deposit #TEST-120256).

Key Outcomes: The main findings of the project are as follows:

• Both forms of the IMRB are reliable, valid, and fair across sex, race, and SES groups of introductory undergraduate biology students (Cromley, Dai, Fechter, Van Boekel, Nelson, & Dane, 2021).
• The measure explains significant variance in course grades, either in combination with SAT/ACT scores or by itself (e.g., if a university has dropped its standardized test requirement)(Cromley, Dai, Fechter, Nelson, Van Boekel, & Du, 2021).
• Students who learn biology and take a 'facts-first' approach such as listing many nouns but not connecting them with verbs, have particularly low learning outcomes (Cromley, Ma, Van Boekel, & Dane, 2020).

Structured Abstract

Setting: This project took place at postsecondary institutions in Pennsylvania and Illinois.

Sample: Participants included undergraduate students enrolled in or who had just completed introductory biology courses. These 2,551 students participated in 5 studies at 2 universities over 3 years. Of these, 211 provided think-aloud or cognitive interview data used to develop and validate the measure. The remaining 2,340 provided multiple-choice response data, used for reliability, validity, and fairness (DIF) analyses.

Research Design and Methods: This project used an iterative measure development and refinement process to create two equivalent versions of our biology reasoning measure. In this process, the team audiotaped students' thinking about biology texts, and identified correct and incorrect conclusions drawn from the biology material. Texts were then shortened, questions were written tapping onto the correct and incorrect reasoning shown by students, and multiple-choice answers were written. The draft measures were given to biology students, who also verbalized their thinking while answering the reasoning questions. These interviews showed that, indeed, the measure requires deductive reasoning but notprior knowledge, test-taking strategies, other learning strategies such as summarizing, or vocabulary. Each version of the multiple-choice measure was then given to large samples of biology students. Analyses suggested a few of the newly developed questions should be dropped; they also showed that both forms met a very high standard for quality measures and could be used to predict student course grades.

Key Measures: In addition to the two forms, the research team collected data on students' background and course grades.

Data Analytic Strategy: Researchers used the 3PLM model for IRT scaling, and the Mantel-Haentzel chi squared test for detecting DIF. They used ESEM with earlier samples and CFA with later samples to provide evidence for unidimensionality. Think-aloud protocols were analyzed using path models, and cognitive interviews were analyzed using non-parametric dependent-samples t tests.

Products and Publications

ERIC Citations: Find available citations in ERIC for this award here.

Publicly available data: Cromley, Jennifer. Inference-Making and Reasoning: Refinement of an Assessment for Use in Gateway Biology Courses, Illinois, 2018–2019. ICPSR38276-v1. Ann Arbor, MI: Inter-university Consortium for Political and Social Research [distributor], 2021–12–07: https://www.icpsr.umich.edu/web/ICPSR/studies/38276/versions/V1. Data are restricted for research use, and requestors will have to comply with ICPSR restrictions to access the files.

Project website: https://www.cromleylab.org/copy-of-meta-analysis-of-multimedia

Select Publications

Journal Articles

Cromley, J. G., Dai, T., Fechter, T. S., Nelson, F. E., Van Boekel, M., & Du, Y. (2021). Development of a tool to assess inference-making and reasoning in biology. Journal of Microbiology and Biology Education, 22(2), e00159-21. https://doi.org/10.1128/jmbe.00159-21

Cromley, J. G., Dai, T., Fechter, T., Van Boekel, M., Nelson, F. E., and Dane, A. (2021). What cognitive interviewing reveals about a new measure of undergraduate biology reasoning. The Journal of Experimental Education, 89(1), 145–168. Full Text

Cromley, J. G., Ma, S., Van Boekel, M., & Dane, N. (2020). Pickup of causal language and inference during and after reading illustrated text. Reading Psychology, 41(3), 157–182. doi.org/10.1080/02702711.2020.1768974 Dai, T., Van Boekel, M., Cromley, J. Nelson, F. and Fechter, T. (2018). Using think-alouds to create a better measure of biology reasoning. SAGE Research Methods Cases [online]. doi: 10.4135/9781526437167 Full Text