Co-Principal Investigator: McLaren, Bruce

Purpose: In this project, researchers will develop and test a tool that offers new ways for middle school teachers to use results from student work in intelligent tutors to facilitate instruction and learning. Data-driven analytics offer potential for providing insight into how students learn in technology-enhanced learning environments. However, less is known about optimizing the design of user interfaces to present analytics to teachers to improve instruction and enhance and personalize student learning.

Project Activities: Researchers will employ an iterative design-based process whereby the team will experiment with different forms of hardware interfaces to organize and present student data to teachers, including augmented reality, heads-up displays, and tablet and mobile apps. This research will inform prototyping of tools to support teacher instruction. After development is completed in year 4, the team will carry out a pilot study in middle-school mathematics classes to measure the effect of the tool on teacher-student interactions, and on student learning processes and outcomes.

Products: Researchers will produce a tool for teachers that will present mathematical data in real time in new formats through the use of mixed reality, heads-up displays, and tablet and mobile apps as students solve complex problems using intelligent tutoring systems. Dissemination activities will include distribution of the tool to educational practitioners as well as producing peer reviewed publications with findings from the project.

Structured Abstract

Setting: The study will take place in middle schools in a large urban district in Florida.

Sample: The study will include 60 grade 7 to 8 mathematics classrooms, approximately 1,200 students, and the teachers in the classrooms.

Intervention: The tool will display mathematical data in real time and in new formats through the use of mixed reality, heads-up displays, and tablet and mobile apps as students solve complex problems using intelligent tutoring systems. The tool will generate information to enhance a teacher’s awareness of who is struggling and aid decision making regarding who to help and how, and will improve teacher-student interactions by providing functionality and easy access to personalized resources to better focus and target such interactions. The team will also develop and publish an Application Programming Interface to allow any learning platform implementing to communicate with the tool. By the end of the project, the tool will be integrated with two commercially available ITS systems, potentially providing the opportunity for widespread use by thousands of math teachers and hundreds thousands of students.

Research Design and Methods: To develop the tool, the researchers will use a design-based implementation procedure where iterations of research and development will occur until feasibility, usability, and learning aims are met. The research team will experiment with multiple hardware interfaces to present data, including mixed reality, heads-up displays, and tablet and mobile apps, to create an orchestration tool. After development is complete, a classroom study with sixty grade 7 and 8 classrooms will examine how the tool affects teacher behaviors and students’ learning processes, and whether it holds promise to improve student learning. Researchers will implement the tool within two existing commercially available tutoring systems which all treatment and control classes will use.

Control Condition: Teachers in the control condition will be provided with a limited-functionality version of the orchestration tool that does not provide real-time recommendations or analytics.

Key Measures: Researchers will gather data using student log data, teacher judgment data, and assessments. The assessments will include multiple-choice and fill-in-the-blank questions that align with Mathematics Florida Standards which are nearly identical to the Common Core State Standards in middle school grades.

Data Analytic Strategies: During the iterative development period, the research team will use observational and log data to understand how expert teachers use results to inform instruction, as well as data mining studies to create recommender models to support awareness and decision making. In the pilot study the research team will use analysis of variance techniques to compare students in the treatment and control conditions. Additionally, researchers will use within-subjects comparisons for all students to test for learning gains by comparing performance at pre- and post-exposure to the intervention, and performance at near and far transfer.

Related IES Projects: Bringing Cognitive Tutors to the Internet: A Website that Helps Middle-School Students Learn Math (R305A080093); Optimizing AI-Based Tutoring Software for Middle-School Mathematics on Smartphones (R305A220386)

Products and Publications

ERIC Citations:Find available citations in ERIC for this award here.

Journal articles

Holstein, K., McLaren, B.M., & Aleven, V. (2019). Co-designing a real-time classroom orchestration tool to support teacher-AI complementarity. Journal of Learning Analytics, 6(2), 27–52. Full text

Proceedings

Fancsali, S. E., Holstein, K., Sandbothe, M., Ritter, S., McLaren, B. M., & Aleven, V. (2020). Towards practical detection of unproductive struggle. In International Conference on Artificial Intelligence in Education (pp. 92–97). Springer, Cham. Full text

Holstein, K., McLaren, B.M., & Aleven, V. (2019, June). Designing for Complementarity: teacher and student needs for orchestration support in ai-enhanced classrooms. In Proceedings of the 20th International Conference on Artificial Intelligence in Education (pp. 157–171). Springer, Cham. Full text

Zhang, C., Huang, Y., Wang, J., Lu, D., Fang, W., Stamper, J., ... & Aleven, V. (2019). Early detection of wheel spinning: comparison across tutors, models, features, and operationalizations. Proceedings of the 12th International Conference on Educational Data Mining Montreal, QC, Canada, July 2–5, 2019.Full text