Co-Principal Investigator: Baker, Ryan S.

Purpose: The purpose of this project is to use data-mining and machine learning methods to explore the relationship between affective and behavioral engagement with measures of student learning within an online adaptive mathematics learning system. Adaptive learning programs for students generate rich data, offering an important opportunity to identify when students are not learning productively and which system features and implementation factors may be most strongly related to productive learning states.

Project Activities: Researchers will first use data mining methods to build automated detectors of engagement based on student behaviors within an online program, LearnBop. The team will collect observations of student engagement in order to validate the computer generated results. In the second set of studies, the research team will collect observational, system logfile, and other extant data to capture system features, classroom implementation factors, and contextual factors. In the final years of the project, the team will complete data analyses, report generation, and other dissemination activities.

Products: The researchers will produce preliminary evidence for how malleable system features and classroom implementation factors predict detectors of engagement and learning, and information describing how engagement may mediate links between malleable system and implementation factors and student learning. Ultimately, the team seeks to generate testable hypotheses about system and implementation improvements that will optimize student engagement and learning.

Structured Abstract

Setting: The project will analyze primary and secondary data from students in Grades 7 and 8 in a large urban Florida public school district.

Sample: The sample includes approximately 1,500 students (750 per grade) in approximately 60 classrooms (30 per grade).

Intervention: LearnBop is an online adaptive math program that offers a variety of scaffolded problems with hints and pop-up instructional videos when students make repeated mistakes.

Control Condition: The nature of the research design does not require a control condition.

Research Design and Methods: In the first phase, the team will use data mining and machine learning to build automated detectors of engagement based on student behaviors within LearnBop and will collect observations of student engagement. During the second phase, the researchers will collect observational, system logfile, and other extant data to examine system features, classroom implementation factors, and contextual factors.

Key Measures: The key measures are automated detectors of engagement, LearnBop system features, classroom implementation factors, contextual factors, and measures of student learning. Within-system measures of student learning are based on student-level performance data on LearnBop problems. Measures of robust learning include scores on state assessments and a study-administered mathematics assessment.

Data Analytic Strategy: Researchers will synchronize system logfiles and field observations to complete analyses related to detector creationby. Within the logfiles, student actions during each problem will be distilled into data features that represent aspects of student behavior typically associated with engagement (e.g., time between clicks and the use of hints). These features will be used to build a detector for each engagement construct (e.g., off-task behavior, confusion, and frustration), using step regression, decision trees, and other algorithms, with ongoing cross-validation analyses. The team will use three-level hierarchical models (problems nested within students nested within teachers) and structural equation modeling to estimate relationships among malleable factors, the detectors of engagement, learning measures, and contextual factors.