Project Activities
The researchers will conduct two implementation studies in which usability, math learning, and qualitative implementation data are used to iteratively refine the developed tool, followed by a pilot study in which the tool will be compared against a business-as-usual control.
Structured Abstract
Setting
The research will be conducted in public fifth and sixth grade classrooms in a diverse suburban school district in California.
Sample
Two implementation studies will be conducted with approximately 60 fifth and sixth grade students in study 1 and 120 students in study 2. The pilot study will include approximately 300 fifth and sixth grade students.
The researchers will develop a scalable instructional technology that could be used to individualize students' engagement in HOT Talk while learning math content, thereby both improving their math learning and training their attention to relations and serving to equalize opportunities for higher order thinking. Previous work incorporated HOT Talk into instructional videos teaching ratio and proportion content interspersed with free-response prompts. This project builds upon that work by developing and testing the effectiveness of training a generative AI chatbot, called a "HOT Mindset Chatbot" to engage students in producing HOT Talk. The chatbot will be trained on relevant math content from a textbook text and on HOT Talk routines through transcripts of everyday HOT Talk and HOT Talk during problem solving tasks.
Research design and methods
This project will begin with an implementation study in which the researchers will use usability, math learning, and qualitative implementation data to iteratively refine the developed tool. In a second implementation study, they will compare the developed digital instruction plus generative AI HOT Mindset Chatbot versus the same digital instruction plus a non-interactive set of scripted HOT Talk questions students will answer. They will use results from study 2 to improve the intervention and update the chatbot if it does not lead to more HOT Talk and learning than the non-interactive questions. Finally, they will run a pilot study of the intervention with 10 classes, comparing the tool against a business-as-usual control.
Control condition
For the implementation studies, there is no control condition. For the pilot study, students will be randomly assigned within classroom to the HOT Mindset Intervention or to a business-as-usual control in which they spend the same allocation of time on the mathematics educational technology currently in use within this school system.
Key measures
All students will be administered the same measures. Key measures include researcher-developed tests of conceptual and procedural knowledge of proportional reasoning, a test of students' spontaneous use of HOT on an unrelated set of problems, an executive function task, and surveys that capture situational interest, mind wandering, and qualitative responses to questions about their experience and the process of participating with the developed intervention.
Data analytic strategy
The research team will explore the relationships between HOT Talk within the transcript of talk between the child and the chatbot, math performance, and engagement (situational interest and mind wandering). The team will also investigate whether individual differences in children's attention to relations are related to learning or whether these relationships are mitigated by engagement with the chatbot. Analyses will use analysis of variance, conventional regression models, and Latent Class Analysis.
Cost analysis strategy
The researchers will evaluate the cost for development as well as implementation of the tool. These relative costs will be considered separately as well as jointly since scaled usage of the technology would not incur the same original development costs. For the development, the cost calculations include costs of personnel to create the video lessons (both researcher and teacher time), time of the videographer, video editor, and programmer. The programmer's time will be committed to training the generative AI chatbot, and the research team to examine the rate of HOT Talk use in the chatbot transcripts. For the implementation, the researchers will include the small-time commitment for teacher training, the cost of teacher time to run the study, the fixed infrastructure costs for computers available to all students, and the opportunity cost of our intervention materials taking the place of other math curricula.
People and institutions involved
IES program contact(s)
Project contributors
Products and publications
The researchers will develop an online HOT Mindset digital training tool, which is a curriculum supplement designed to be administered in a blended in-class format. The project will also result in peer-reviewed publications and presentations as well as additional dissemination products that reach education stakeholders such as practitioners and policymakers.
Publications:
ERIC Citations: Find available citations in ERIC for this award here.
Related projects
Questions about this project?
To answer additional questions about this project or provide feedback, please contact the program officer.
