Project Activities
Utilizing extensive data from two premier math initiatives - MATHia and UF Data Lagoon project, which include billions of log data, hundreds of millions of assessment results, and millions of genuine math discussions – researchers will advance existing generative AI research to develop a low-cost, multimodal, and ethical teachable agent. In the exploratory phase, researchers will collaborate with 10 middle school math teachers from varied backgrounds to co-design the agent and gauge 100 teachers' experiences with third-party (e.g., MagicSchool AI) and the research team’s generative AI learning tools via an online professional development program. Subsequently, researchers will use this refined generative AI-powered agent to conduct three studies for development, innovation, and impact: (1) three rounds of usability studies to identify both student and teacher usage, perceptions, and feedback related to MAGICAL-Math; (2) two rounds of feasibility studies to identify and address technology solution challenges; and (3) a randomized controlled trial to systematically investigate the effects of MAGICAL-Math on student learning outcomes.
Structured Abstract
Setting
Public middle schools (Grades 6-8) from three partner school districts in Florida that serve a large population of racial/ethnic minority and low-income students.
Sample
The project targets underserved middle school students. Usability studies will involve 3 teachers and 225 students and each feasibility study will include 5 teachers and 375 students. The randomized controlled trial (RCT) will engage 46 teachers and 3,450 students over a 16-week semester. All studies will involve three partner school districts in Florida, where over 50% of students are racial/ethnic minorities and more than 55% qualify for free or reduced-price meals.
Research design and methods
First, researchers will co-design the generative AI technology with 10 teachers and develop professional development materials based on feedback from 100 teachers about using generative AI in classrooms. Then, researchers will focus on usability and feasibility, with at least eight teachers integrating the generative AI teachable agent into their classes and reporting any issues. Then, researchers will conduct a three-level cluster randomized trial with randomization at the school level, with one teacher per school. Teachers will commit to weekly use of MAGICAL-Math with their students over a 16-week intervention. Researchers will ensure high fidelity through system logs and regular communication.
Control condition
Both control and treatment groups will have access to MATHia. The control group follows a business-as-usual approach, using MATHia at the same frequency as the treatment group but without the support from MAGICAL-Math.
Key measures
The Florida Assessment of Student Thinking will be the distal measure to assess the summative mathematics performance of middle school students. The team will use MATHia's Adaptive Personalized Learning Score (APLSE) to assess formative math performance, including sections mastered per hour, total section mastery percent, and mastery level. Researchers will use the Student Interest in Mathematics Scale, a validated and reliable instrument, to assess students' mathematical interests, and will use students' Interactive Log data to measure growth in students' mathematics engagement across the intervention period.
Data analytic strategy
Researchers will assess students' experiences with the generative AI tool using validated scales for usability and acceptability, stealth assessment via logs, focused group interviews, and students' instructional quality and success status. For teacher evaluation, researchers will combine quantitative survey analysis and qualitative interviews. In the RCT, researchers will examine MAGICAL-Math's impact on student interest and achievement using a three-level model and will evaluate the effect of MAGICAL-Math on changes in student engagement over 16 weeks through four-level growth curve models.
Cost analysis strategy
The cost study will follow a three-phase strategy to determine the costs associated with implementing the MAGICAL-Math intervention. Researchers will use the ingredients method to estimate the total cost of implementing the intervention for all treatment schools in the RCT and the average cost per school and student. The cost analysis will use a societal perspective to better inform decision-making at the local and state level.
People and institutions involved
IES program contact(s)
Project contributors
Products and publications
Publications:
Find available citations in ERIC for this award here.
Questions about this project?
To answer additional questions about this project or provide feedback, please contact the program officer.
