Project Activities
MathSpring is a fully developed online intervention that is in widespread use in schools around the country. Results from pilot studies of MathSpring demonstrated its promise for improving student learning in mathematics, and provided a basis for a larger-scale, independent, and rigorous efficacy evaluation. Year 1 served as a practice year, during which treatment group teachers received professional development to practice using MathSpring to supplement their mathematics instruction. As well in Year 1, the data collection instruments were piloted, refined, and finalized, and implementation fidelity metrics were developed. In Years 2-4, the researchers implemented the intervention in treatment classrooms, and researchers measured the mathematics achievement of 5th- and 6th-grade students in both treatment and control groups as well as student dispositions and emotions in mathematics. Years 5 and 6 focused on data analysis and dissemination of study findings. Researchers analyzed efficacy using a hierarchical linear regression (HLM) model that examines mean differences in achievement between students in treatment and control groups, controlling for prior achievement and other covariates.
Structured Abstract
Setting
The researchers recruited a representative sample of schools in Massachusetts and ensured the sample included low-performing schools and schools with lower-socioeconomic status student bodies.
Sample
The study team recruited 80 5th or 6th grade teachers in the state of Massachusetts for participation across three cohorts. Sixty-four of these teachers participated in the study.
MathSpring is a supplemental online mathematics intervention where students are presented with and solve problems customized to their individual level. MathSpring generates hints if students need assistance. The intervention provides affective support through animated pedagogical agents which share messages designed to promote growth mindset and facilitate reflection to track progress and set goals. A teacher dashboard presents diagnostic reports to facilitate reviews of students' progress, enabling teachers to adapt instruction accordingly. Teachers also receive professional development to support their use of the reports as a formative assessment tool to shape classroom practices.
Research design and methods
The research team employed a clustered randomized design to answer a set of research questions about the primary outcome, moderators, and implementation fidelity. Teachers were blocked by school and randomly assigned to either the treatment or control condition. Teachers and students in their math classes participated in the study for a full school year.
Control condition
Teachers assigned to the business-as-usual control condition continued using existing instructional practices and supplemental technologies (other than MathSpring).
Key measures
The primary measures of student mathematics achievement included the Grades 5 and 6 Massachusetts Comprehensive Assessment System (MCAS) math assessment. The Mathematics Diagnostic Testing Project (MDTP) Grade 6 and 7 Readiness Tests (MRT) served as a supplemental measure. Student’s dispositions and emotions in math was measured by a survey composed of multiple validated scales on students’ attitude and mindset towards math and interest in math. Implementation fidelity in treatment classrooms was assessed through researcher-developed instructional logs and system usage records in MathSpring.
Data analytic strategy
The research team addressed the impact question using a two-level hierarchical linear regression model of mean differences in Grades 5 and 6 achievements between students in two conditions, controlling for prior achievement and other covariates. They used moderator analyses to examine the impact of the intervention on the learning of students with low-baseline mathematics achievement, and different demographic backgrounds. In addition, they conducted exploratory analyses to examine the link between student learning outcomes and implementation fidelity.
Cost analysis strategy
Researchers determined the costs associated with implementing the intervention using the "ingredients method." This entailed systematically collecting and analyzing all expenditures on personnel, facilities, equipment, materials, and training.
Key outcomes
The main findings of this project, as reported by the principal investigator (Feng, et al., 2025), are as follows:
- No statistically significant differences in mathematics achievement were observed between the treatment and control groups on both MCAS test and the MDTP tests.
- The study did not detect differential treatment impacts for the subgroups of interests, including students with different levels of prior achievement or different demographic characteristics.
- The study found no evidence of a significant difference between the treatment and control groups was observed for students’ approaches towards learning or disposition or affect towards math.
- Descriptive analysis showed that the amount of time spent in MathSpring ranged from 2 -978 minutes (mean = 190, median = 150, standard deviation = 157 minutes).
- Analyses showed that students in classrooms of high usage outperformed students in the control group on the MCAS posttest performance and students in the medium and low usage groups were comparable with that of students in the control group, controlling for pretest.
The total cost of the MathSpring intervention in this efficacy study was estimated to be $109,935. The incremental cost of MathSpring intervention relative to a business-as-usual control group in this efficacy study is estimated to be $47,496, resulting in an incremental cost per student of $48.
People and institutions involved
IES program contact(s)
Project contributors
Products and publications
The products of this project include evidence of the efficacy of the MathSpring program, conference presentations and peer reviewed publications, and publicly available reports for school administrators.
Study registration:
Publications:
ERIC Citations: Find available citations in ERIC for this award here.
Journal Article:
Feng, M., Brezack, N., Huang, K., Kao, Y., Collins, K., Lee, M., Schneider, M., & Chan, W. (2025). Evaluating the Efficacy of an Intelligent Tutoring System that Integrates Affect Supports into Math Learning. Journal of Computer Assisted Learning. DOI: 10.1111/jcal.70106
Book Chapter:
Feng, M., Brezack, N., Huang, K., Schneider, M., Arroyo, I., Woolf. B., Allessio, D. (2025). AI-Powered Math Learning: Evaluating the Impact of a Personalized Tutoring Platform that Responds to Affect. In: Cristea, A.I., Walker, E., Lu, Y., Santos, O.C., Isotani, S. (eds) Artificial Intelligence in Education. AIED 2025. Communications in Computer and Information Science, vol 2591. Springer, Cham. pp. 118-125. https://doi.org/10.1007/978-3-031-99264-3_15
Conference Proceedings:
Brezack, N., Chan, W., & Feng, M. (2024). Pandemic-Related Perseverance During Math Problem-Solving in MathSpring, an Educational Technology Platform that Responds to Student Affect. In Lindgren, R., Asino, T. I., Kyza, E. A., Looi, C. K., Keifert, D. T., & Suarez, E. (Eds.), Proceedings of the 18th International Conference of the Learning Sciences - ICLS 2024 (pp. 650-657). International Society of the Learning Sciences. https://doi.org/10.22318/icls2024.147455.
Brezack, N., Chan, W., & Feng, M. (2024). Student Effort and Progress Learning Analytics Data Inform Teachers’ SEL Discussions in Math Class. In LAK '24: Proceedings of the 14th Learning Analytics and Knowledge Conference. Pp. 338-348. Kyoto, Japan. March, 2024. https://doi.org/10.1145/3636555.3636888.
Brezack, N., Lee, M., Collins, K., Chan, W., & Feng, M. (2025). Uncovering Student Profiles with Problem Solving and Effort Data from a Tutoring System. In Proceedings of the Twelfth ACM Conference on Learning @ Scale (L@S ’25), July 21-23, 2025, Palermo, Italy. ACM, New York, NY, USA.336–340. https://doi.org/10.1145/3698205.3733951
Feng, M., Brezack, N., Schneider, M., Collins, K., Chan, W., & Lee, M. (2024). Lessons Learned from a Research-to-Practice Scale-Up of an Adaptive Math Learning Platform. Proceedings of the Eleventh ACM Conference on Learning@ Scale (pp. 456-460). Atlanta, GA. July 2024.
Lee, M., Huang, C.-W., Collins, K., Feng, M. (2025). Examining the Relationship between Math Anxiety, Effort, and Learning Outcomes Using Multilevel Latent Class Analysis. LAK '25: Proceedings of the 15th International Learning Analytics and Knowledge Conference. pp. 227-237. https://doi.org/10.1145/3706468.37064
Questions about this project?
To answer additional questions about this project or provide feedback, please contact the program officer.
