Co-Principal Investigators: Feng, Mingyu; Bang, Hee Jin; Thai, Khanh-Phuong

Purpose: Mathematics education, especially at early grade levels, continues to be an important focus for national education improvement efforts, and more districts are introducing technology as part of their solution. The researchers will conduct an initial study to generate evidence of the efficacy of My Math Academy, a widely used, game-based, supplemental online curriculum that enables personalized learning through embedded assessments, adaptive learning trajectories, and a performance dashboard that helps teachers monitor student progress and personalize learning. My Math Academy has promising results from pilot studies which provide a basis for a larger-scale, independent, rigorous study of the efficacy of the intervention for improving student learning outcomes in kindergarten and first grade mathematics.

Project Activities: The study will randomly assign 66 elementary schools into treatment and control conditions. Kindergarten classrooms and first grade classrooms in treatment schools will implement My Math Academy as a supplemental mathematics program in 2023–2024 and 2024–2025, respectively; whereas control schools will use their business-as-usual supplemental mathematics activities.

Products: This study will shine new light on the understanding of implementing widely used technology-based programs in authentic school settings. Research dissemination will focus on how to implement a technology-based early math program, which provides personalized content embedded assessments, and performance dashboards for teachers; and to what extent the program can improve learning outcomes for kindergarten and first grade students when implemented under routine conditions in authentic school settings. The researchers will produce peer-reviewed publications and a final publicly accessible dataset.

Structured Abstract

Setting: The study will take place in schools in California.

Sample: The study sample will include 66 elementary schools involving approximately 132 teachers and 1,980 students in kindergarten and first grade, respectively. The study is aligned with California's focus on using personalized learning and leverage the availability of and support for technology in the state. The researchers will seek to recruit a sample of schools in California that includes low-performing schools and schools with large populations of students from lower socioeconomic status backgrounds.

Intervention: Teachers and students in kindergarten and first grade will use My Math Academy as a supplemental curriculum to learn and practice mathematics skills. Teachers will receive real-time reports to facilitate their review of students' usage and progress and adapt instruction accordingly. Students will work through personalized learning pathways with interactive, game-like, story-based learning activities that are customized to their individual needs. Students will receive instruction on math concepts and skills, scaffolding and feedback to help solve difficult problems, and embedded assessments to gauge their understanding of individual concepts and skills. Teachers will receive professional development to support their use of student data and the program as a tool to personalize classroom practices.

Research Design and Methods: Researchers will use a clustered randomized experimental design to answer a set of research questions about the primary outcomes, moderators, and implementation fidelity. They randomly assign schools to either the treatment or control condition and will ask schools to participate for two full school years (kindergarten: 2023–2024 and first grade: 2024–2025). Data on implementation fidelity and contrast between conditions will be collected during both years. Cost data will be gathered systematically during the study.

Control Condition: Teachers assigned to the business-as-usual control condition will continue using existing supplemental mathematics activities (other than My Math Academy).

Key Measures: The Test of Early Mathematics Ability, third edition (TEMA) will be the primary measure of student early mathematics skills.

Data Analytic Strategy: Researchers will assess efficacy using a three-level hierarchical linear regression model of mean differences in TEMA scores between students in two conditions, controlling for prior achievement and other covariates. They will use moderator analyses to examine the impact of the intervention on the learning of students with low-baseline mathematics achievement, and different demographic backgrounds. The team will carry out mediator analyses to examine the link between student learning outcomes and mediating variables such as engagement in mathematics learning. They will also complete exploratory analyses to address the accumulated impact of multi-year My Math Academy exposure on student outcomes. In addition, they will examine cost and cost-effectiveness of My Math Academy.

Cost Analysis: The team will systematically gather cost data during the study. The researchers will use the ingredients method to gather costs systemically and estimate total costs of implementing My Math Academy during the study. The team will also estimate incremental costs per student above and beyond the control condition and calculate cost-effectiveness ratios for the treatment using effect sizes of student performance on the primary outcome measure.