Project Activities
First, the project team will train a large language model and engage in prompt engineering so that CAIT can interact with students through speech and text as they work through solving math problems. Next, the team will conduct design-based implementation research, usability testing, fairness testing, and feasibility testing to inform the iterative refinement of CAIT. Finally, the team will conduct a pilot study to establish evidence of promise for CAIT in supporting students' math learning and engagement.
Structured Abstract
Setting
Participating schools are located in urban, suburban, and rural locations in Massachusetts, Pennsylvania, Idaho, Louisiana, and Iowa.
Sample
In the first phase of the work, 10 middle school students will participate in 2 cycles of usability testing for CAIT as it is being developed. Next, 10 middle school math teachers and their students will participate in 2 rounds of feasibility studies to help identify logistical issues and inform iterative development. Across all 3 years of the project, 25 teachers from diverse backgrounds and schools will join fairness testing to systematically spot-check the content from the development and review some of the full dialogues for bias, ethics, correctness, and safety. Finally, 20 teachers and their 1,500 students will be recruited to participate in the pilot study.
CAIT will engage in human-like dialogue with students through a conversational interface powered by natural language processing that uses both text-to-speech and speech-to-text capabilities, allowing students to interact with CAIT through speech or text. The conversational interface ensures that students can ask questions, seek explanations, and receive personalized feedback in a natural and engaging manner. CAIT will respond to student queries, provide explanations, and offer personalized feedback in a conversational format and will adapt to each student's performance level and learning pace allows for personalization comparable to human tutoring. This self-paced learning approach reduces the pressure of keeping up with the class or feeling left behind while empowering students to take charge of their education and build a strong foundation in math. CAIT will also provide continuous assessment and adaptive assignments by assessing students' understanding and misconceptions in real-time through dialogue and problem-solving, identifying areas of strengths and weakness, and then determining additional problems for students to solve that reflect appropriate prerequisite skills.
The project team will integrate CAIT within ASSISTments, which is freely available, and will be able to function on low-cost devices, mobile phones, and low-profile computers with Internet access to ensure that students have equal opportunities to access CAIT.
Research design and methods
First, the project team along with the teacher design team will develop and iteratively refine CAIT. The project team will train a large language model and engage in prompt engineering so that CAIT can interact with students through speech and text as they work through solving math problems. Next, the project team will conduct usability, fairness, and feasibility testing with students and collect data via system backend logs, interviews, focus groups, and surveys. Findings from the studies will be used to iteratively refine and improve the product. Finally, the team will conduct the pilot study to establish evidence of promise for enhancing math learning and engagement in math problem-solving. The pilot study will use a cluster-randomized control trial design, randomly assigning classrooms to use CAIT within ASSISTments to support homework or to use ASSISTments without access to CAIT.
Control condition
No control condition exists for the usability, fairness, or feasibility studies. For the pilot study, the classrooms assigned to the control condition will continue using ASSISTments but will not have access to CAIT.
Key measures
The primary outcome measure is students' performance on the end-of-unit summative assessments from the Illustrative Mathematics textbook (assigned by teachers). The team will also measure students' growth mindset, grit, and attitudes toward math. Using system log data, the project team will calculate students' math time-on-task.
Data analytic strategy
The project team will transcribe and analyze qualitative data from interviews, focus groups, and open-end survey responses using a linear, hierarchical analysis approach. For the pilot study, they will perform an intent-to-treatment analysis using a 2-level hierarchical linear model to account for classroom-level clustering. The team will conduct moderation analysis to estimate the effects on students of low socioeconomic status and low prior-year achievement. They will also utilize learning-analytic techniques to devise machine learning-based models on ASSISTments log data to identify patterns in the learning process data.
Cost analysis strategy
The project team will gather cost data systematically throughout the study. The team will use the ingredients method to gather costs and estimate the total costs of implementing CAIT during the pilot study.
People and institutions involved
IES program contact(s)
Project contributors
Partner institutions
WestEd
Products and publications
The project team plans to disseminate research findings and information about developed products to teachers through the existing channels at the ASSISTments Foundation. The team will also disseminate findings to education researchers, educational technology developers, and the broader academic community through publications, presentations at academic and practitioner conferences, and the project website.
Publications:
ERIC Citations: Find available citations in ERIC for this award here.
Related projects
Supplemental information
Partner Institutions: The ASSISTments Foundation; WestEd; Greater Commonwealth Virtual School
Questions about this project?
To answer additional questions about this project or provide feedback, please contact the program officer.
