|Title:||Optimizing AI-Based Tutoring Software for Middle-School Mathematics on Smartphones|
|Principal Investigator:||Aleven, Vincent||Awardee:||Carnegie Mellon University|
|Program:||Cognition and Student Learning [Program Details]|
|Award Period:||4 years (07/01/2022 – 06/30/2026)||Award Amount:||$2,000,000|
|Type:||Development and Innovation||Award Number:||R305A220386|
Co-Principal Investigators: Koedinger, Kenneth R.; Carvalho, Paulo; Sewall, Jonathan
Purpose: In this project, researchers will conduct design-based research with students, parents, and teachers to develop a mobile data-optimized tutoring system to promote deliberate math practice for middle-school students, combined with a new system for parent-student motivational nudges to encourage parents to support their children in doing homework. The research team will optimize units from the Mathtutor platform—a comprehensive web-based tutoring system for middle-school mathematics—and specifically focus on its algebra strand. Doing math is critical to mathematics learning, and teacher-assigned projects and homework provide important opportunities for learning-by-doing. Many students are underserved because of unfavorable conditions for doing homework with timely and on-target support. This project will test research-based hypotheses to determine how student math outcomes improve under more favorable homework conditions.
Project Activities: During years 1–3 of the project, the research team will conduct user-testing and design-based research to create mobile tutors and teacher-parent nudges in an integrated design. Concurrently, during years 1–4, approximately 130 sixth to eighth grade students will participate in 5 experimental studies to evaluate the effects of the intervention on student math outcomes.
Products: The research team will generate new scientific knowledge regarding how best to use advanced learning technologies to support effective homework for middle school math. Researchers will disseminate project information and findings through reports and peer-reviewed publications to reach relevant educational and scientific communities. If successful, the project will result in a proof-of-concept of combining (1) data-optimized deliberate practice supported by AI-based tutoring software, (2) delivery of the tutoring software on smartphones, and (3) social/motivational support through teacher-parent nudges to improve student math outcomes.
Setting: For experiments 1–4, the research will take place primarily in schools that are ethnically and racially diverse with a high percentage of low-SES students. For experiment 5, the research also will take place in schools that are primarily White and have a high percentage of high-SES students to provide an SES control.
Sample: During years 1–3 of the project, sixth to eighth grade students, parents, and teachers will participate in user-centered research and co-design to create mobile tutors and teacher-parent nudges in an integrated design. Concurrently, during Years 1–4, approximately 130 sixth to eighth grade students will participate in 5 experimental studies to evaluate the effects of the intervention on student math outcomes.
Intervention: The project team will test the synergistic effect of three interventions intended to create more favorable conditions for homework: (1) data-optimized deliberate practice supported by AI-based tutoring software, (2) delivery of the tutoring software on smartphones, with an offline mode so that students can work on their phones even when having no connectivity for long periods of time, and (3) social/motivational support through teacher-parent nudges to help students stay motivated. As a platform for the research, the research team will use the free-access Mathtutor platform, which hosts AI-based tutoring software with comprehensive mathematics content for middle school.
Research Design and Methods: The research team will conduct five experiments to test the separate and joint effects of the interventions to create favorable homework conditions. The experiments will be interleaved with the design and development activities. For all five experiments, students will be randomly assigned within-class using a two-unit planned crossover design—Group A will receive the smartphone and tutor treatment for unit 1 and Group B will receive the treatment for unit 2—and counterbalanced order across subjects.
Control Condition: The control condition teachers will deliver homework without technology support. Students in the control condition will be asked to solve fixed problem sets on paper, without immediate individualized feedback or adaptive mastery learning.
Key Measures: The research team will measure student mastery of key skills in targeted mathematics areas using both researcher-developed items and standardized test items (including items from the Pennsylvania System of School Assessment (PSSA) and the National Assessment of Educational Progress (NAEP)). The research team will measure student engaged time with Mathtutor and the number of opportunities to practice knowledge components based on tutor log data.
Data Analytic Strategy: The research team will use hierarchical linear models with students nested within classes within teachers to compare learning gains between conditions.
Cost Analysis: The research team will conduct a cost analysis following the "ingredients" method to identify, quantify, and price (including time costs) each of the elements required for the intervention and the control conditions. The research team will calculate the cost-effectiveness of the intervention by calculating the costs associated with the intervention considering student gains.
Related IES Projects: Bringing Cognitive Tutors to the Internet: A Website that Helps Middle-School Students Learn Math (R305A080093); Enhancing Student Learning with an Orchestration Tool for Personalized Teacher-Student Interactions in Classrooms Using Intelligent Tutoring Software Education Technology (R305A180301)