|Title:||Inference-Making and Reasoning: Refinement of an Assessment for Use in Gateway Biology Courses|
|Principal Investigator:||Cromley, Jennifer||Awardee:||University of Illinois, Urbana-Champaign|
|Program:||Postsecondary and Adult Education [Program Details]|
|Award Period:||4 years (9/1/2016-8/31/2020)||Award Amount:||$756,527|
Co-Principal Investigators: Ting Dai (Temple University), Tia Sukin (Pacific Metrics Corporation), Lawrence Rome (University of Pennsylvania)
Purpose: In order to help researchers, postsecondary institutions, and educators better identify and assist students struggling in gateway biology courses, this team of researchers will develop an assessment of students' scientific reasoning ability. Scientific reasoning and the ability to generate inferences predict student achievement and retention in undergraduate introductory biology courses. Students who cannot draw such inferences are more likely to struggle in their courses and are less likely to persist in STEM majors. Thus, having a validated assessment to identify students struggling with scientific reasoning and inference-drawing skills may help programs and educators provide supports to at-risk students.
Project Activities: Using an existing assessment they developed for research purposes, the researchers will conduct a series of studies to refine the existing items and test their validity and reliability. The research team will first analyze existing data collected in previous research that used the existing assessment, follow with mixed method work to assist the researchers better understand items that are not functioning as intended and revise them as needed, and then complete analyses to ensure the validity and appropriate use of the full assessment.
Products: The final products include a refined measure of inference making and reasoning in postsecondary, introduction to biology courses and peer-reviewed publications.
Setting: Research sites include large state universities in Pennsylvania and in Illinois.
Sample: For Study 1, the researchers will use existing data collected over 7 years from approximately 1,800 students (primarily freshmen and sophomores) enrolled in introductory biology courses. For Studies 2 and 3, they will collect data from approximately 1,400 students enrolled in similar courses.
Assessment: The assessment that will be refined and validated focuses on measuring the types of reasoning and inference-making that are critical for a deep understanding of science-related course material and that play a vital role in transfer of learning to new contexts. In particular, the assessment focuses on students' ability to demonstrate applied reasoning with recently presented information. The existing 20-minute-long, multiple-choice assessment presents students with brief passages and questions that require them to reason using biological principals with new material. For example, they read about the immune system, which is not typically covered in the introductory courses, and choose from the possible responses. The incorrect responses reflect common, specific reasoning errors students often make in the introductory biology courses. The researchers have used the existing assessment for their research and have found strong Cronbach's alpha reliability but have not fully validated it using psychometric approaches.
Research Design and Methods: Using secondary and primary data, the researchers will conduct three studies aimed to refine an existing assessment of inference-making and reasoning in biology and ensure its validity and reliability. In Study 1, the researchers will use existing data collected from students in biology courses. They will calibrate items and test them for differential functioning by students' race, sex, and socio-economic status. In Study 2, they will conduct cognitive interviews with a new sample of students to understand why certain items are not functioning as intended and will make modifications accordingly. Study 3 has three parts. In part one, they will conduct tests of validity and reliability, item calibration using a multidimensional model, and tests for differential item functioning. In part two, they will use a measurement decision theory (Bayesian) approach to set a cut score that can identify students at high risk of course failure, while controlling for students' ACT Reading scores and High School GPA. In part three, they will determine whether the assessment is effective for documenting student growth over a semester.
Control Condition: Due to the nature of this research, there is no control or comparison condition.
Key Measures: In addition to student responses to the assessment, the researchers will collect course grades, scores on a final exam or project (which is included in calculation of grades), students' intention to remain in a STEM major, and demographics (e.g., age, sex, race, parental education/socioeconomic status, major, self-reported ACT Reading score).
Data Analytic Strategy: The researchers will use multiple analytic strategies including confirmatory factor analysis, item response theory, and differential item functioning to ensure the validity and appropriate use of the full assessment.
Cromley, J. G., Dai, T., Fechter, T., Van Boekel, M., Nelson, F. E., and Dane, A. (2019). What Cognitive Interviewing Reveals About a New Measure of Undergraduate Biology Reasoning. The Journal of Experimental Education, 1-24. ED594569
Dai, T., Van Boekel, M., Cromley, J. Nelson, F. and Fechter, T. (2018). Using Think-Alouds to Create a Better Measure of Biology Reasoning. SAGE Research Methods Cases [online]. doi: 10.4135/9781526437167 Full Text