Co-Principal Investigator(s): Heffernan, Cristina; Feng, Mingyu

Partner Institution(s): The ASSISTments Foundation, WestEd, Greater Commonwealth Virtual School

Purpose: The purpose of this project is to develop a conversational artificial intelligence (AI) tutor (CAIT- pronounced as "Kate") to support independent math practice for middle school students who struggle with math and, otherwise, may not have access to after-school tutoring. CAIT will be integrated into ASSISTments, an existing, freely available, and evidence-based online math platform with widely used homework assignments from open education resources (OER). Through this work, the project team aims to dramatically improve students' engagement and math learning during independent math problem-solving time and narrow the persistent learning gap between groups of students by expanding the reach of after-school tutoring to low-income students who could not afford it otherwise.

Project Activities: First, the project team will train a large language model and engage in prompt engineering so that CAIT can interact with students through speech and text as they work through solving math problems. Next, the team will conduct design-based implementation research, usability testing, fairness testing, and feasibility testing to inform the iterative refinement of CAIT. Finally, the team will conduct a pilot study to establish evidence of promise for CAIT in supporting students' math learning and engagement.

Products: The project team plans to disseminate research findings and information about developed products to teachers through the existing channels at the ASSISTments Foundation. The team will also disseminate findings to education researchers, educational technology developers, and the broader academic community through publications, presentations at academic and practitioner conferences, and the project website.

Structured Abstract

Setting: Participating schools are located in urban, suburban, and rural locations in Massachusetts, Pennsylvania, Idaho, Louisiana, and Iowa.

Sample:  In the first phase of the work, 10 middle school students will participate in 2 cycles of usability testing for CAIT as it is being developed. Next, 10 middle school math teachers and their students will participate in 2 rounds of feasibility studies to help identify logistical issues and inform iterative development. Across all 3 years of the project, 25 teachers from diverse backgrounds and schools will join fairness testing to systematically spot-check the content from the development and review some of the full dialogues for bias, ethics, correctness, and safety. Finally, 20 teachers and their 1,500 students will be recruited to participate in the pilot study.

Project Focus: Technology Product: CAIT will engage in human-like dialogue with students through a conversational interface powered by natural language processing that uses both text-to-speech and speech-to-text capabilities, allowing students to interact with CAIT through speech or text. The conversational interface ensures that students can ask questions, seek explanations, and receive personalized feedback in a natural and engaging manner. CAIT will respond to student queries, provide explanations, and offer personalized feedback in a conversational format and will adapt to each student's performance level and learning pace allows for personalization comparable to human tutoring. This self-paced learning approach reduces the pressure of keeping up with the class or feeling left behind while empowering students to take charge of their education and build a strong foundation in math. CAIT will also provide continuous assessment and adaptive assignments by assessing students' understanding and misconceptions in real-time through dialogue and problem-solving, identifying areas of strengths and weakness, and then determining additional problems for students to solve that reflect appropriate prerequisite skills.

The project team will integrate CAIT within ASSISTments, which is freely available, and will be able to function on low-cost devices, mobile phones, and low-profile computers with Internet access to ensure that students have equal opportunities to access CAIT.

Research Design and Methods: First, the project team along with the teacher design team will develop and iteratively refine CAIT. The project team will train a large language model and engage in prompt engineering so that CAIT can interact with students through speech and text as they work through solving math problems. Next, the project team will conduct usability, fairness, and feasibility testing with students and collect data via system backend logs, interviews, focus groups, and surveys. Findings from the studies will be used to iteratively refine and improve the product. Finally, the team will conduct the pilot study to establish evidence of promise for enhancing math learning and engagement in math problem-solving. The pilot study will use a cluster-randomized control trial design, randomly assigning classrooms to use CAIT within ASSISTments to support homework or to use ASSISTments without access to CAIT.

Control Condition: No control condition exists for the usability, fairness, or feasibility studies. For the pilot study, the classrooms assigned to the control condition will continue using ASSISTments but will not have access to CAIT.

Key Measures: The primary outcome measure is students' performance on the end-of-unit summative assessments from the Illustrative Mathematics textbook (assigned by teachers). The team will also measure students' growth mindset, grit, and attitudes toward math. Using system log data, the project team will calculate students' math time-on-task.

Data Analytic Strategy: The project team will transcribe and analyze qualitative data from interviews, focus groups, and open-end survey responses using a linear, hierarchical analysis approach. For the pilot study, they will perform an intent-to-treatment analysis using a 2-level hierarchical linear model to account for classroom-level clustering. The team will conduct moderation analysis to estimate the effects on students of low socioeconomic status and low prior-year achievement. They will also utilize learning-analytic techniques to devise machine learning-based models on ASSISTments log data to identify patterns in the learning process data.

Cost Analysis: The project team will gather cost data systematically throughout the study. The team will use the ingredients method to gather costs and estimate the total costs of implementing CAIT during the pilot study.

Related IES Projects: Using Web-Based Cognitive Assessment Systems for Predicting Student Performance on State Exams (R305K030140), Making Longitudinal Web-Based Assessments Give Cognitively Diagnostic Reports to Teachers, Parents, and Students While Employing Mastery Learning (R305A070440), An Efficacy Study of Online Mathematics Homework Support: An Evaluation of the ASSISTments Formative Assessment and Tutoring Platform (R305A120125), Efficacy of ASSISTments Online Homework Support for Middle School Mathematics Learning: A Replication Study (R305A170641), Revisions to the ASSISTments Digital Learning Platform to Expand Its Support for Rigorous Education Research (R305N210049)