Co-Principal Investigators: Bhat, Suma; Anderson, Carolyn; Angrave, Lawrence; Bosch, Nigel

Purpose:In this project, the researchers explored interactions among various characteristics of online instruction and postsecondary students' success in STEM courses. In particular, they explored how students who are traditionally underrepresented in STEM—namely, first-generation students, and students from minoritized groups—benefit from or are impeded by online features, such as participation in discussion boards. Online instruction has the potential to make course content more accessible to large numbers of students, thereby strengthening the STEM pipeline. However, instructors and administrators need to know if certain online pedagogical content or approaches create barriers so that they can provide better advising and supports to students who are take online courses or (re)design online courses to serve all students. In general, the researchers found that the features of courses that have been shown to lead to student success in in-person college STEM courses, such as active participation in discussions and asking for help, were the same features that led to student success in online college STEM courses. Importantly, this was true independent of whether the students came from groups traditionally underrepresented in STEM.

Project Activities:The researchers used data mining, qualitative, and statistical analytic approaches to explore correlations among various course features (such as if and how students were required to participate in discussion forums), student behaviors (such as asking for help in the discussion forum), student background factors (such as first-generation college student) and students' success (such as the rate of improvement and final grade) in the course.

Key Outcomes:The main findings of this exploratory study are as follows:

• Students from groups underrepresented in STEM (UR-STEM) improved at the same rate (Bosch et al., 2019) or at greater rates (Williams-Dobosz, Azevedo, et al., 2021) than students who are not UR-STEM.
• Women — a group underrepresented in many STEM fields, and in the particular courses investigated in this project — did not use gendered language differently than men in their discussion forum posts, namely both men and women used stereotypical male approaches and female approaches similarly (Henricks et al., 2020; 2021).
• There was a significant positive relationship between seeking help on class discussion forums and improvement (Williams-Dobosz, Azevedo, et al., 2021). Moreover, students who requested help in course discussion forums were responded to equally, regardless of how explicitly they appealed for help, and UR-STEM students requested for and responded to help similarly but received help at greater rates than their non-UR-STEM peers (Williams-Dobosz, Jeng, et al., 2021).

Structured Abstract

Setting:The research was conducted at a large, midwestern university, which is located in a small urban community (population = ~150,000), though during the pandemic, students enrolled at this university participated from across the globe.

Sample:The sample included undergraduate students enrolled in large online STEM classes. The researchers used data from 2535 students. In analyses where gender was included, 931 students identified as women. In analyses where race or ethnicity was included, 179 students identified as belonging to a racially or ethnically minoritized group.

Malleable Factors:In this project, the researchers identified malleable factors that are indicated in the Community of Inquiry theoretical framework. In particular, they examined the constraints and requirements of the course structure and several forms of student participation in the class.

Research Design and Methods:The researchers used data mining techniques, qualitative analysis, and statistical approaches to explore connections between student background factors, course structures (such as the number and type of required posts to the course discussion forums), and student behaviors, to predict student success in the course. They conducted a series of studies to explore different correlations and different factors.

Control Condition:Due to the exploratory nature of this project, there was no control or comparison condition.

Key Measures:The researchers used administrative data that students reported when applying for admission to assess student background characteristics (such as gender, underrepresentation in STEM, first-generation college status) and obtained measures of student success (such as grades, improvement in the course) from university records. They used data obtained from online course logs for measures of student engagement with online course content (such as time-sequenced clickstream interaction data, discussion board use) and researcher-developed coding rubrics for measures of student participation (such as using gendered language or seeking help in the discussion forum).

Data Analytic Strategy:The researchers used multiple modeling strategies, including hierarchical cluster analysis, logistic regression, and proportional odds models.

Products and Publications

ERIC Citations:Find available citations in ERIC for this award here.

Project Website:https://publish.illinois.edu/ilearngroup/

Additional Online Resources and Information:

• Extracting and analyzing data from learning management systems is complex, so the researchers in collaboration with Dr. Stuart Raeburn (Michigan State University) developed code and documentation for this process and made it publicly available: https://github.com/classtranscribe/
• The researchers also developed a program to anonymize data automatically: https://github.com/pnb/forum-anonymization
• The researchers also shared code to help automate tagging of metacognitive language in texts: https://github.com/aigagror/metacognitive_phrase_detector

Select Publications:

Journal Articles

Henricks, G. M., Bhat, S., & Perry, M. (2021). Gender and gendered discourse in two online STEM college courses. Computer-Based Learning in Context, 3(1), 1–16.

Williams-Dobosz, D., Jeng, A., Azevedo, R. Bosch, N., Ray, C., & Perry, M. (2021). Ask for help: Online help-seeking and help-giving as indicators of cognitive and social presence for students underrepresented in chemistry. Journal of Chemical Education, 98, 3693–3703.

Proceedings

Angrave L., Jensen K., Zhang Z., Mahipal C., Mussulman M., Schmitz C., Baird, R., Liu H., Sui R., Wu M., & Kooper R. (2020). Improving student accessibility, equity, course performance, and lab skills: How introduction of ClassTranscribe is changing engineering education at the University of Illinois. Proceedings of ASEE Annual Virtual Conference2020. Full text

Angrave, L., Zhang Z., Henricks G., & Mahipal C. (2020). Who benefits? Positive learner outcomes from behavioral analytics of online lecture video viewing using ClassTranscribe. SIGCSE'20: Proceedings of the 51st ACM Technical Symposium on Computer Science Education(pp. 1193–1199). Portland, OR: ACM. Full Text

Bosch, N., Crues, R. W., Shaik, N., & Paquette, L. (2020). "Hello, [REDACTED]": Protecting student privacy in analyses of online discussion forums. Proceedings of the 13th International Conference on Educational Data Mining (EDM 2020)(pp. 39–49). International Educational Data Mining Society. Full text

Bosch, N., Huang, E., Angrave, L.C., & Perry, M. (2019). Modeling improvement for underrepresented minorities in online STEM education. In G.A. Papadopoulos, G. Samaras, S. Weibelzahl, D. Jannach, and O. Santos (Eds.), Proceedings of the 27th ACM Conference on User Modeling, Adaptation and Personalization(pp. 327–335).New York, NY: ACM. Full Text

Henricks, G., Perry, M., & Bhat, S. (2020). Gender and gendered discourse in two online STEM courses. In M. Gresalfi and I. S. Horn (Eds.), Proceedings of the 14th International Conference on Learning Sciences (ICLS) 2020, Vol 2.(pp. 811–812). Nashville, TN: International Society of the Learning Sciences.

Huang, E., Valdiviejas, H., & Bosch, N. (2019). I'm sure! Automatic detection of metacognition in online course discussion forums. Proceedings of the 8th International Conference on Affective Computing and Intelligent Interaction (ACII 2019), 241–247. Full Text

Jay, V., Henricks, G., Bosch, N., Perry, M., Bhat, S. P., Williams-Dobosz, D., Angrave, L. C., & Shaik, N. (2020). Online discussion forum help-seeking behaviors of students underrepresented in STEM. In M. Gresalfi and I. S. Horn (Eds.), Proceedings of the 14th International Conference on Learning Sciences (ICLS) 2020, Vol 2.(pp. 809–810). Nashville, TN: International Society of the Learning Sciences.

Jeng, A., Valdiviejas, H., & Perry, M. (2022). A path analysis of gender differences in social presence in online course discussion forums.To appear in the Proceedings of the International Conference on Learning Sciences (ICLS) 2022. Hiroshima, Japan: International Society of the Learning Sciences.

Valdiviejas, H., & Bosch, N. (2020). Using association rule mining to uncover rarely occurring relationships in two university online STEM courses: A comparative analysis. Proceedings of the 13th International Conference on Educational Data Mining (EDM 2020)(pp. 686–690). International Educational Data Mining Society. Full text

Williams-Dobosz, D., Azevedo, R., Jeng, A., Thakkar, V., Bhat, S., Bosch, N., & Perry, M. (2021). A social network analysis of online engagement for college students traditionally underrepresented in STEM. In M. Scheffel, N. Dowell, S. Joksimovic, and G. Siemens (Eds.) The impact we make: The contributions of learning analytics to learning,Proceedings of the 11th International Conference on Learning Analytics and Knowledge(LAK21) (pp. 207–215). New York, NY: ACM.

Zhou, J. & Bhat, S. (2021). Modeling consistency of engagement patterns in online courses. In M. Scheffel, N. Dowell, S. Joksimovic, and G. Siemens (Eds.) The impact we make: The contributions of learning analytics to learning, Proceedings of the 11th International Conference on Learning Analytics and Knowledge (LAK21) (pp. 226–236). New York, NY: ACM. Full text

Zhou, J., Zeng, Z., Gong, H. & Bhat, S. (2022). Idiomatic expression paraphrasing without strong supervision. In the Thirty-Sixth AAAI Conference on Artificial Intelligence (AAAI-22) Proceedings, 2022,pp. 11774–11782.