Skip Navigation
Funding Opportunities | Search Funded Research Grants and Contracts

IES Grant

Title: Bootstrapping Achievement and Motivation in STEM: An Integrated Cognitive-Motivational Intervention to Improve Biology Grades
Center: NCER Year: 2014
Principal Investigator: Cromley, Jennifer Awardee: Temple University
Program: Postsecondary and Adult Education      [Program Details]
Award Period: 4 years (8/1/2014-7/31/2018) Award Amount: $1,379,250
Type: Development and Innovation Award Number: R305A140602
Description:

Co-Principal Investigators: Kaplan, Avi; Balsai, Michael; Perez, Tony

Purpose: First-year biology college courses constitute a gateway for success in multiple STEM majors, yet many academically able students struggle to learn the concepts and fail to perceive them as relevant to their future career. To support biology instruction across diverse higher education settings, the researchers created a set of online instructional supports that 'wrap around' existing courses to enhance students' learning, motivation, and achievement in first-year college biology courses.

Project Activities: The research team developed a suite of very brief web-based supports, which were delivered to students via the Blackboard Learning Management System along the semester as supplements to faculty regular instruction. This suite included brief videos and messages that targeted central cognitive and motivational processes. During the last phase of the project, the team carried out a pilot test of the final suite of supports.

Key Outcomes: The main features of the curriculum and findings of the project's pilot study are as follows:

  • Key features of the curriculum include study strategy instruction videos, worked examples videos, costs offsetting videos, self-efficacy enhancing feedback, and relevance writing assignments.
  • Student grades increased by an average of 6.6 percentage points (equivalent to a one grade band [e.g., from a C+ to a B-] to two grade bands [e.g., from a B to an A-] increase)
  • Participation levels were high, suggesting that students found the supports useful
  • In one condition the combination of self-efficacy enhancing messages with strategy instruction videos) at one university over 3 semesters, the achievement of students from under-represented minority (URM) caught up with the achievement of majority peers. Analyses of participant feedback suggested that, in this condition, URM students were more likely to take ownership of their own learning, and less likely to blame difficult exams for their achievement.

Structured Abstract

Setting: This project took place at postsecondary institutions in Pennsylvania, Illinois, and Virginia.

Sample: Participants included undergraduate students enrolled in introductory biology courses. These 3,092 students participated in 10 studies at 3 universities over 4 years. Of these, 2,122 were involved in the iterative development phase (in 7 studies over 3 years), and 970 from 3 universities were involved in the year 4 pilot phase. Across all 10 studies, participants were 64 percent female; 30 percent Asian, 13 percent Black, 6 percent Latino/a, 41 percent White, and 8 percent of self-identified other or mixed race. They had a mean age of 19.5, and 42 percent were freshmen, 31 percent sophomores, 18 percent juniors, and 6 percent seniors. Using the definition of neither parent with a bachelor's degree, 34 percent were first-generation college students. Mean standardized college admissions test scores were relatively high, with SAT math M = 600, SAT reading M = 580, ACT math M = 28.9, and ACT reading M = 29.1. No information was collected about disability status of participants.

Intervention: The piloted intervention included 5 components: one of two cognitive supports plus one of three motivational supports. The cognitive supports included (1) videotaped modeling of specific study strategies (drawing to learn, comparing and contrasting drawings, making concept maps, etymology, and summarizing) and (2) videotaped worked examples of short, medium-difficulty problems. The motivational supports included (1) relevance writing tasks, in which students write a series of brief essays regarding the relevance of the biology content for their learning, major, career, and life goals; (2) targeted self-efficacy-enhancing feedback intended to build student self-efficacy; and (3) targeted videotaped messages to offset students' perceived sacrifices when completing the biology course. The rationale behind the intervention is that students need better study skills—as shown by successful previous interventions—and students benefit from motivational supports—again, as shown by successful previous interventions—but higher motivation by itself may not be enough unless students know how to study, and study skills may be in a student's repertoire but not used unless there is motivation to enact them. All components were delivered via Blackboard and used in conjunction with typical classroom instruction (e.g., lectures).

Research Design and Methods: This project used an iterative design process to develop the curriculum. In this process, the team tried out and refined various combinations of six instructional components (one of three cognitive plus one of three motivational support strategies) and tested them in comparison to one another to determine which components should be included in the final suite of supports. Analyses of the effects of different conditions, of student feedback, and of the timing of student access of the supports were used to inform removing and retaining supports (e.g., dropping lecture videos), and modifying delivery to enhance student use of the supports. For example, the studies showed that students benefited more from the interventions when they used the supports throughout the course, rather than cramming at the end. To encourage this behavior, students received credit when they accessed the supports in a timely matter (e.g., when relevant to the material being covered). Analyses also informed adjustments to the wording of supports, including clarification of the guides for relevance writing and adding written captions to all videos. These improved versions of the supports were implemented in the pilot versions of the intervention, which tested strategy instruction + relevance vs. strategy instruction + self-efficacy support vs. business as usual control (pretest-posttest only), with credit for accessing supports in a timely manner.

Control Condition: In all studies, the control condition was a pretest-posttest only control group that received usual classroom instruction and no extra supports.

Key Measures: Across the development phase and the pilot phase, key outcome measures included course exams and final course grades. In addition, the research team collected data on the extent and timing of students' access of the support (e.g., number of relevance writings that were completed, and when). Researchers also used questionnaires at the beginning and end of the semesters to assess changes in students' self-efficacy, valuing of gateway courses, and perceived non-monetary costs (e.g., foregoing study abroad, time with family, and/or psychological burden of being a biology student).

Data Analytic Strategy: Researchers used ANOVAs and Dunnett planned comparisons to test the effect of each intervention on grades against the business-as-usual control group. This approach was used during the iterative design stage and the pilot studies. No adjustments were needed for nested data, as each course was taught by one instructor in one lecture at each university. Researchers used logistic regressions to test the effect of each intervention on intention to stay in STEM vs. intention to leave STEM against the business-as-usual control group; this approach was used during the iterative design stage and the pilot studies. Researchers used secondary analyses to test whether motivation and deductive reasoning scores mediate the effects of conditions on grades. These were tested using path analysis. Finally, the overall results of all studies across 4 years at 3 universities were analyzed using meta-analysis.

Products

ERIC Citations:  Find available citations in ERIC for this award here.

Publicly available data: Sufficient statistics for each condition in each experiment are provided here.

Project Website: https://www.cromleylab.org/

Select Publications:

Journal articles

Cromley, J.G., Perez, T., & Kaplan, A. (2016). Undergraduate STEM achievement and retention cognitive, motivational, and institutional factors and solutions. Policy Insights From the Behavioral and Brain Sciences, 3(1): 4–11. Full text.

Cromley, J. G., Perez, T., Kaplan, A., Dai, T., Mara, K., & Balsai, M. J. (2020). Combined cognitive-motivational modules delivered via an LMS increase undergraduate biology grades. Technology, Mind, and Behavior, 1(2). doi.org/10.1037/tmb0000020 Full text

Kaplan, A., Cromley, J. G., Perez, T., Dai, T., Mara, K. R., & Balsai, M. J. (2020). The role of context in educational RCT findings: A call to redefine "evidence-based practice." Educational Researcher, 49(4), 285–288. Full text

Perez, T., Dai, T. Kaplan, A., Cromley, J. G., Brooks, W. D., White, A. C., Mara, K. R., & Balsai, M. J. (2019). Interrelations among expectancies, task values, and perceived costs in undergraduate biology achievement. Learning and Individual Differences, 72, 26–38. Full text

Proceedings

Cromley, J. G., & Mara, K. (2018). Comparing and contrasting within diagrams: An effective study strategy. In P. Chapman, G. Stapleton, A. Moktefi, S. Perez-Kriz, & F. Bellucci (Eds.) Diagrams 2018, LNAI 10871, pp. 492–499. doi.org/10.1007/978-3-319-91376-6_44


Back