Project Activities
The researchers used data mining, qualitative, and statistical analytic approaches to explore correlations among various course features (such as if and how students were required to participate in discussion forums), student behaviors (such as asking for help in the discussion forum), student background factors (such as first-generation college student) and students' success (such as the rate of improvement and final grade) in the course.
Structured Abstract
Setting
The research was conducted at a large, midwestern university, which is located in a small urban community (population = ~150,000), though during the pandemic, students enrolled at this university participated from across the globe.
Sample
The sample included undergraduate students enrolled in large online STEM classes. The researchers used data from 2535 students. In analyses where gender was included, 931 students identified as women. In analyses where race or ethnicity was included, 179 students identified as belonging to a racially or ethnically minoritized group.
In this project, the researchers identified malleable factors that are indicated in the Community of Inquiry theoretical framework. In particular, they examined the constraints and requirements of the course structure and several forms of student participation in the class.
Research design and methods
The researchers used data mining techniques, qualitative analysis, and statistical approaches to explore connections between student background factors, course structures (such as the number and type of required posts to the course discussion forums), and student behaviors, to predict student success in the course. They conducted a series of studies to explore different correlations and different factors.
Control condition
Due to the exploratory nature of this project, there was no control or comparison condition.
Key measures
The researchers used administrative data that students reported when applying for admission to assess student background characteristics (such as gender, underrepresentation in STEM, first-generation college status) and obtained measures of student success (such as grades, improvement in the course) from university records. They used data obtained from online course logs for measures of student engagement with online course content (such as time-sequenced clickstream interaction data, discussion board use) and researcher-developed coding rubrics for measures of student participation (such as using gendered language or seeking help in the discussion forum).
Data analytic strategy
The researchers used multiple modeling strategies, including hierarchical cluster analysis, logistic regression, and proportional odds models.
Key outcomes
The main findings of this exploratory study are as follows:
- Students from groups underrepresented in STEM (UR-STEM) improved at the same rate (Bosch et al., 2019) or at greater rates (Williams-Dobosz, Azevedo, et al., 2021) than students who are not UR-STEM.
- Women — a group underrepresented in many STEM fields, and in the particular courses investigated in this project — did not use gendered language differently than men in their discussion forum posts, namely both men and women used stereotypical male approaches and female approaches similarly (Henricks et al., 2020; 2021).
- There was a significant positive relationship between seeking help on class discussion forums and improvement (Williams-Dobosz, Azevedo, et al., 2021). Moreover, students who requested help in course discussion forums were responded to equally, regardless of how explicitly they appealed for help, and UR-STEM students requested for and responded to help similarly but received help at greater rates than their non-UR-STEM peers (Williams-Dobosz, Jeng, et al., 2021).
People and institutions involved
IES program contact(s)
Project contributors
Products and publications
ERIC Citations: Find available citations in ERIC for this award here.
Additional Online Resources and Information:
- Extracting and analyzing data from learning management systems is complex, so the researchers in collaboration with Dr. Stuart Raeburn (Michigan State University) developed code and documentation for this process and made it publicly available: https://github.com/classtranscribe/
- The researchers also developed a program to anonymize data automatically: https://github.com/pnb/forum-anonymization
- The researchers also shared code to help automate tagging of metacognitive language in texts: https://github.com/aigagror/metacognitive_phrase_detector
Select Publications:
Journal Articles
Henricks, G. M., Bhat, S., & Perry, M. (2021). Gender and gendered discourse in two online STEM college courses. Computer-Based Learning in Context, 3(1), 1–16.
Williams-Dobosz, D., Jeng, A., Azevedo, R. Bosch, N., Ray, C., & Perry, M. (2021). Ask for help: Online help-seeking and help-giving as indicators of cognitive and social presence for students underrepresented in chemistry. Journal of Chemical Education, 98, 3693–3703.
Proceedings
Angrave L., Jensen K., Zhang Z., Mahipal C., Mussulman M., Schmitz C., Baird, R., Liu H., Sui R., Wu M., & Kooper R. (2020). Improving student accessibility, equity, course performance, and lab skills: How introduction of ClassTranscribe is changing engineering education at the University of Illinois. Proceedings of ASEE Annual Virtual Conference2020. Full text
Angrave, L., Zhang Z., Henricks G., & Mahipal C. (2020). Who benefits? Positive learner outcomes from behavioral analytics of online lecture video viewing using ClassTranscribe. SIGCSE'20: Proceedings of the 51st ACM Technical Symposium on Computer Science Education(pp. 1193–1199). Portland, OR: ACM. Full Text
Bosch, N., Crues, R. W., Shaik, N., & Paquette, L. (2020). "Hello, [REDACTED]": Protecting student privacy in analyses of online discussion forums. Proceedings of the 13th International Conference on Educational Data Mining (EDM 2020)(pp. 39–49). International Educational Data Mining Society. Full text
Bosch, N., Huang, E., Angrave, L.C., & Perry, M. (2019). Modeling improvement for underrepresented minorities in online STEM education. In G.A. Papadopoulos, G. Samaras, S. Weibelzahl, D. Jannach, and O. Santos (Eds.), Proceedings of the 27th ACM Conference on User Modeling, Adaptation and Personalization(pp. 327–335).New York, NY: ACM. Full Text
Henricks, G., Perry, M., & Bhat, S. (2020). Gender and gendered discourse in two online STEM courses. In M. Gresalfi and I. S. Horn (Eds.), Proceedings of the 14th International Conference on Learning Sciences (ICLS) 2020, Vol 2.(pp. 811–812). Nashville, TN: International Society of the Learning Sciences.
Huang, E., Valdiviejas, H., & Bosch, N. (2019). I'm sure! Automatic detection of metacognition in online course discussion forums. Proceedings of the 8th International Conference on Affective Computing and Intelligent Interaction (ACII 2019), 241–247. Full Text
Jay, V., Henricks, G., Bosch, N., Perry, M., Bhat, S. P., Williams-Dobosz, D., Angrave, L. C., & Shaik, N. (2020). Online discussion forum help-seeking behaviors of students underrepresented in STEM. In M. Gresalfi and I. S. Horn (Eds.), Proceedings of the 14th International Conference on Learning Sciences (ICLS) 2020, Vol 2.(pp. 809–810). Nashville, TN: International Society of the Learning Sciences.
Jeng, A., Valdiviejas, H., & Perry, M. (2022). A path analysis of gender differences in social presence in online course discussion forums.To appear in the Proceedings of the International Conference on Learning Sciences (ICLS) 2022. Hiroshima, Japan: International Society of the Learning Sciences.
Valdiviejas, H., & Bosch, N. (2020). Using association rule mining to uncover rarely occurring relationships in two university online STEM courses: A comparative analysis. Proceedings of the 13th International Conference on Educational Data Mining (EDM 2020)(pp. 686–690). International Educational Data Mining Society. Full text
Williams-Dobosz, D., Azevedo, R., Jeng, A., Thakkar, V., Bhat, S., Bosch, N., & Perry, M. (2021). A social network analysis of online engagement for college students traditionally underrepresented in STEM. In M. Scheffel, N. Dowell, S. Joksimovic, and G. Siemens (Eds.) The impact we make: The contributions of learning analytics to learning,Proceedings of the 11th International Conference on Learning Analytics and Knowledge(LAK21) (pp. 207–215). New York, NY: ACM.
Zhou, J. & Bhat, S. (2021). Modeling consistency of engagement patterns in online courses. In M. Scheffel, N. Dowell, S. Joksimovic, and G. Siemens (Eds.) The impact we make: The contributions of learning analytics to learning, Proceedings of the 11th International Conference on Learning Analytics and Knowledge (LAK21) (pp. 226–236). New York, NY: ACM. Full text
Zhou, J., Zeng, Z., Gong, H. & Bhat, S. (2022). Idiomatic expression paraphrasing without strong supervision. In the Thirty-Sixth AAAI Conference on Artificial Intelligence (AAAI-22) Proceedings, 2022,pp. 11774–11782.
Project website:
Publications:
ERIC Citations: Find available citations in ERIC for this award here.
Select Publications:
Journal Articles
Henricks, G. M., Bhat, S., & Perry, M. (2021). Gender and gendered discourse in two online STEM college courses. Computer-Based Learning in Context, 3(1), 1-16.
Williams-Dobosz, D., Jeng, A., Azevedo, R. Bosch, N., Ray, C., & Perry, M. (2021). Ask for help: Online help-seeking and help-giving as indicators of cognitive and social presence for students underrepresented in chemistry. Journal of Chemical Education, 98, 3693-3703.
Proceedings
Angrave L., Jensen K., Zhang Z., Mahipal C., Mussulman M., Schmitz C., Baird, R., Liu H., Sui R., Wu M., & Kooper R. (2020). Improving student accessibility, equity, course performance, and lab skills: How introduction of ClassTranscribe is changing engineering education at the University of Illinois. Proceedings of ASEE Annual Virtual Conference2020. Full text
Angrave, L., Zhang Z., Henricks G., & Mahipal C. (2020). Who benefits? Positive learner outcomes from behavioral analytics of online lecture video viewing using ClassTranscribe. SIGCSE'20: Proceedings of the 51st ACM Technical Symposium on Computer Science Education(pp. 1193-1199). Portland, OR: ACM. Full Text
Bosch, N., Crues, R. W., Shaik, N., & Paquette, L. (2020). "Hello, [REDACTED]": Protecting student privacy in analyses of online discussion forums. Proceedings of the 13th International Conference on Educational Data Mining (EDM 2020)(pp. 39-49). International Educational Data Mining Society. Full text
Bosch, N., Huang, E., Angrave, L.C., & Perry, M. (2019). Modeling improvement for underrepresented minorities in online STEM education. In G.A. Papadopoulos, G. Samaras, S. Weibelzahl, D. Jannach, and O. Santos (Eds.), Proceedings of the 27th ACM Conference on User Modeling, Adaptation and Personalization(pp. 327-335).New York, NY: ACM. Full Text
Henricks, G., Perry, M., & Bhat, S. (2020). Gender and gendered discourse in two online STEM courses. In M. Gresalfi and I. S. Horn (Eds.), Proceedings of the 14th International Conference on Learning Sciences (ICLS) 2020, Vol 2.(pp. 811-812). Nashville, TN: International Society of the Learning Sciences.
Huang, E., Valdiviejas, H., & Bosch, N. (2019). I'm sure! Automatic detection of metacognition in online course discussion forums. Proceedings of the 8th International Conference on Affective Computing and Intelligent Interaction (ACII 2019), 241-247. Full Text
Jay, V., Henricks, G., Bosch, N., Perry, M., Bhat, S. P., Williams-Dobosz, D., Angrave, L. C., & Shaik, N. (2020). Online discussion forum help-seeking behaviors of students underrepresented in STEM. In M. Gresalfi and I. S. Horn (Eds.), Proceedings of the 14th International Conference on Learning Sciences (ICLS) 2020, Vol 2.(pp. 809-810). Nashville, TN: International Society of the Learning Sciences.
Jeng, A., Valdiviejas, H., & Perry, M. (2022). A path analysis of gender differences in social presence in online course discussion forums.To appear in the Proceedings of the International Conference on Learning Sciences (ICLS) 2022. Hiroshima, Japan: International Society of the Learning Sciences.
Valdiviejas, H., & Bosch, N. (2020). Using association rule mining to uncover rarely occurring relationships in two university online STEM courses: A comparative analysis. Proceedings of the 13th International Conference on Educational Data Mining (EDM 2020)(pp. 686-690). International Educational Data Mining Society. Full text
Williams-Dobosz, D., Azevedo, R., Jeng, A., Thakkar, V., Bhat, S., Bosch, N., & Perry, M. (2021). A social network analysis of online engagement for college students traditionally underrepresented in STEM. In M. Scheffel, N. Dowell, S. Joksimovic, and G. Siemens (Eds.) The impact we make: The contributions of learning analytics to learning,Proceedings of the 11th International Conference on Learning Analytics and Knowledge(LAK21) (pp. 207-215). New York, NY: ACM.
Zhou, J. & Bhat, S. (2021). Modeling consistency of engagement patterns in online courses. In M. Scheffel, N. Dowell, S. Joksimovic, and G. Siemens (Eds.) The impact we make: The contributions of learning analytics to learning, Proceedings of the 11th International Conference on Learning Analytics and Knowledge (LAK21) (pp. 226-236). New York, NY: ACM. Full text
Zhou, J., Zeng, Z., Gong, H. & Bhat, S. (2022). Idiomatic expression paraphrasing without strong supervision. In the Thirty-Sixth AAAI Conference on Artificial Intelligence (AAAI-22) Proceedings, 2022,pp. 11774-11782.
Additional project information
Additional Online Resources and Information:
- Extracting and analyzing data from learning management systems is complex, so the researchers in collaboration with Dr. Stuart Raeburn (Michigan State University) developed code and documentation for this process and made it publicly available: https://github.com/classtranscribe/
- The researchers also developed a program to anonymize data automatically: https://github.com/pnb/forum-anonymization
- The researchers also shared code to help automate tagging of metacognitive language in texts: https://github.com/aigagror/metacognitive_phrase_detector
Questions about this project?
To answer additional questions about this project or provide feedback, please contact the program officer.
