Skip Navigation

Cognition and Student Learning

Grantees

- OR -

Investigator

- OR -

Goals

- OR -

FY Awards

- OR -

A Theory and Data Driven Approach for Identifying Evidence of Collaborative Problem Solving Skills

Year: 2017
Name of Institution:
Educational Testing Service (ETS)
Goal: Exploration
Principal Investigator:
Andrews, Jessica
Award Amount: $1,399,250
Award Period: 3 years (07/01/2017 – 06/30/2020)
Award Number: R305A170432

Description:

Co-Principal Investigators: Tanner Jackson, Sidney D'Mello (University of Notre Dame), Gregory Chung (University of California, Los Angeles)

Purpose: The purpose of this project is to develop a framework to assess and identify collaborative problem-solving (CPS) skills in computer-based educational environments and to explore the relationship between collaborative problem solving and student learning outcomes. Collaborative problem solving is recognized as a key 21st-century competency, however, no valid and reliable tool exists to measure and assess the influence of collaboration on educational outcomes. This research aims to address three key questions: (1) what is the CPS construct in general and in the paradigm of computerized learning environments, and what student actions and behaviors constitute evidence; (2) what is the nature of the association between CPS and student learning outcomes, particularly in computerized educational environments; and (3) how can a model be developed to analyze and automate assessment of collaborative problem solving skills.

Project Activities: In Year 1, the research team will develop a top-down theory driven concept ontology of student collaborative behavior. In Year 2, the research team will use the ontology to develop a computational model that can enable automated detection of evidence of collaboration from data captured in middle school students' interactions with computerized educational environments. In Year 3, the research team will conduct a pilot test of the model to explore the relationship between middle school students' CPS skills and student learning outcomes.

Products: Researchers will produce a theoretical framework would support the future development of interventions and assessments of complex skills including collaboration, communication and team work.  The research team will also produce  peer-reviewed publications.

Structured Abstract

Setting: Participating public and charter middle schools will be located in urban and suburban areas of New York and New Jersey.

Sample: Approximately 360 middle school students will participate in this research.

Intervention: Due to the exploratory nature of this project, there is no intervention. The malleable factor of interest is collaborative problem-solving, and it will be studied in the context of two fully developed computerized educational environments, Mars Generation One: Argubot Academy and Tetralogue. This project will result in a theoretical framework that would support the future development of interventions and assessments of complex skills including collaboration, communication, and team work.

Research Design and Methods: In Year 1, the research team will develop a top-down theory driven concept ontology of student collaborative behavior. In Year 2, the research team will use the ontology and data collected from a study with middle school students to develop a computational model that can enable automated detection of evidence of collaboration from data captured in middle school students' interactions with computerized educational environments. For the Year 2 study, students will be randomly assigned at the classroom level to play either Tetralogue or Mars Generation One: Argubot Academy, and will play the game with a randomly assigned classmate as part of regular classroom instruction. As students interact with their assigned game, the research team will collect multimodal data, including audio/visual information, keystroke data, chat box input, and game state information. In Year 3, the research team will conduct a pilot test of the computational model to explore the relationship between middle school students' CPS skills and educational outcomes. On the first day of the study, students in the study complete a background questionnaire. Following the questionnaire, students will be randomly assigned to pairs and assigned to the Tetralogue task. On the second day, students will be randomly assigned into new pairs and assigned to the Mars Generation One: Argubot Academy task. For both tasks, process data and multimodal data will be captured as students play. Following completion of the second task, students will take a posttest of their science knowledge.

Control Condition: Due to the exploratory nature of the research design, there is no control condition.

Key Measures: Key measures include video data, keystroke data, chat box input, game state information, and within-game assessments of science knowledge, argumentation skills, and CPS skills.

Data Analytic Strategy: The research team will use a variety of different analysis techniques, including data mining and machine learning techniques, such as cluster analysis and supervised learning (specifically, Support Vector Machines and Random Forests) to develop the computational model of CPS skills. The research team will validate the accuracy of the model by comparing its predictions to CPS-ontology-based, human-provided annotations. They will establish validity of generalizability to new students by splitting the data set by students and repeatedly building models and testing them on new students using cross-validation. In addition, the research team will use regression models to determine the association between CPS factors and student learning outcomes.