Project Activities
In Year 1, the research team will develop a top-down theory driven concept ontology of student collaborative behavior. In Year 2, the research team will use the ontology to develop a computational model that can enable automated detection of evidence of collaboration from data captured in middle school students' interactions with computerized educational environments. In Year 3, the research team will conduct a pilot test of the model to explore the relationship between middle school students' CPS skills and student learning outcomes.
Structured Abstract
Setting
Participating public and charter middle schools will be located in urban and suburban areas of New York and New Jersey.
Sample
Approximately 360 middle school students will participate in this research.
Due to the exploratory nature of this project, there is no intervention. The malleable factor of interest is collaborative problem-solving, and it will be studied in the context of two fully developed computerized educational environments, Mars Generation One: Argubot Academy and Tetralogue. This project will result in a theoretical framework that would support the future development of interventions and assessments of complex skills including collaboration, communication, and team work.
Research design and methods
In Year 1, the research team will develop a top-down theory driven concept ontology of student collaborative behavior. In Year 2, the research team will use the ontology and data collected from a study with middle school students to develop a computational model that can enable automated detection of evidence of collaboration from data captured in middle school students' interactions with computerized educational environments. For the Year 2 study, students will be randomly assigned at the classroom level to play either Tetralogue or Mars Generation One: Argubot Academy, and will play the game with a randomly assigned classmate as part of regular classroom instruction. As students interact with their assigned game, the research team will collect multimodal data, including audio/visual information, keystroke data, chat box input, and game state information. In Year 3, the research team will conduct a pilot test of the computational model to explore the relationship between middle school students' CPS skills and educational outcomes. On the first day of the study, students in the study complete a background questionnaire. Following the questionnaire, students will be randomly assigned to pairs and assigned to the Tetralogue task. On the second day, students will be randomly assigned into new pairs and assigned to the Mars Generation One: Argubot Academy task. For both tasks, process data and multimodal data will be captured as students play. Following completion of the second task, students will take a posttest of their science knowledge.
Control condition
Due to the exploratory nature of the research design, there is no control condition.
Key measures
Key measures include video data, keystroke data, chat box input, game state information, and within-game assessments of science knowledge, argumentation skills, and CPS skills.
Data analytic strategy
The research team will use a variety of different analysis techniques, including data mining and machine learning techniques, such as cluster analysis and supervised learning (specifically, Support Vector Machines and Random Forests) to develop the computational model of CPS skills. The research team will validate the accuracy of the model by comparing its predictions to CPS-ontology-based, human-provided annotations. They will establish validity of generalizability to new students by splitting the data set by students and repeatedly building models and testing them on new students using cross-validation. In addition, the research team will use regression models to determine the association between CPS factors and student learning outcomes.
People and institutions involved
IES program contact(s)
Project contributors
Products and publications
Researchers will produce a theoretical framework would support the future development of interventions and assessments of complex skills including collaboration, communication and team work. The research team will also produce peer-reviewed publications.
Publications:
Amon, M. J., Vrzakova, H., & D'Mello, S. K. (2019). Beyond dyadic coordination: Multimodal behavioral irregularity in triads predicts facets of collaborative problem solving. Cognitive science, 43(10), e12787.
Andrews‐Todd, J., Jackson, G. T., & Kurzum, C. (2019). Collaborative problem solving assessment in an online mathematics task. ETS Research Report Series, 2019(1), 1-7.
D'Mello, S., Stewart, A. E., Amon, M. J., Sun, C., Duran, N. D., & Shute, V. (2019, January). Towards Dynamic Intelligent Support for Collaborative Problem Solving. In TTW@ AIED (pp. 59-65).
Stewart, A., & D’Mello, S. K. (2018). Connecting the dots towards collaborative AIED: Linking group makeup to process to learning. In Artificial Intelligence in Education: 19th International Conference, AIED 2018, London, UK, June 27–30, 2018, Proceedings, Part I 19 (pp. 545-556). Springer International Publishing.
Subburaj, S. K., Stewart, A. E., Ramesh Rao, A., & D'Mello, S. K. (2020, October). Multimodal, multiparty modeling of collaborative problem solving performance. In Proceedings of the 2020 International Conference on Multimodal Interaction (pp. 423-432).
Vrzakova, H., Amon, M. J., Stewart, A., Duran, N. D., & D'Mello, S. K. (2020, March). Focused or stuck together: multimodal patterns reveal triads' performance in collaborative problem solving. In Proceedings of the tenth international conference on learning analytics & knowledge (pp. 295-304).
Questions about this project?
To answer additional questions about this project or provide feedback, please contact the program officer.
