Project Activities
The study will randomly assign 66 elementary schools into treatment and control conditions. Kindergarten classrooms and first grade classrooms in treatment schools will implement My Math Academy as a supplemental mathematics program in 2023–2024 and 2024–2025, respectively; whereas control schools will use their business-as-usual supplemental mathematics activities.
Structured Abstract
Setting
The study will take place in schools in California.
Sample
The study sample will include 66 elementary schools involving approximately 132 teachers and 1,980 students in kindergarten and first grade, respectively. The study is aligned with California's focus on using personalized learning and leverage the availability of and support for technology in the state. The researchers will seek to recruit a sample of schools in California that includes low-performing schools and schools with large populations of students from lower socioeconomic status backgrounds.
Teachers and students in kindergarten and first grade will use My Math Academy as a supplemental curriculum to learn and practice mathematics skills. Teachers will receive real-time reports to facilitate their review of students' usage and progress and adapt instruction accordingly. Students will work through personalized learning pathways with interactive, game-like, story-based learning activities that are customized to their individual needs. Students will receive instruction on math concepts and skills, scaffolding and feedback to help solve difficult problems, and embedded assessments to gauge their understanding of individual concepts and skills. Teachers will receive professional development to support their use of student data and the program as a tool to personalize classroom practices.
Research design and methods
Researchers will use a clustered randomized experimental design to answer a set of research questions about the primary outcomes, moderators, and implementation fidelity. They randomly assign schools to either the treatment or control condition and will ask schools to participate for two full school years (kindergarten: 2023–2024 and first grade: 2024–2025). Data on implementation fidelity and contrast between conditions will be collected during both years. Cost data will be gathered systematically during the study.
Control condition
Teachers assigned to the business-as-usual control condition will continue using existing supplemental mathematics activities (other than My Math Academy).
Key measures
The Test of Early Mathematics Ability, third edition (TEMA) will be the primary measure of student early mathematics skills.
Data analytic strategy
Researchers will assess efficacy using a three-level hierarchical linear regression model of mean differences in TEMA scores between students in two conditions, controlling for prior achievement and other covariates. They will use moderator analyses to examine the impact of the intervention on the learning of students with low-baseline mathematics achievement, and different demographic backgrounds. The team will carry out mediator analyses to examine the link between student learning outcomes and mediating variables such as engagement in mathematics learning. They will also complete exploratory analyses to address the accumulated impact of multi-year My Math Academy exposure on student outcomes. In addition, they will examine cost and cost-effectiveness of My Math Academy.
Cost analysis strategy
The team will systematically gather cost data during the study. The researchers will use the ingredients method to gather costs systemically and estimate total costs of implementing My Math Academy during the study. The team will also estimate incremental costs per student above and beyond the control condition and calculate cost-effectiveness ratios for the treatment using effect sizes of student performance on the primary outcome measure.
People and institutions involved
IES program contact(s)
Project contributors
Products and publications
This study will shine new light on the understanding of implementing widely used technology-based programs in authentic school settings. Research dissemination will focus on how to implement a technology-based early math program, which provides personalized content embedded assessments, and performance dashboards for teachers; and to what extent the program can improve learning outcomes for kindergarten and first grade students when implemented under routine conditions in authentic school settings. The researchers will produce peer-reviewed publications and a final publicly accessible dataset.
Questions about this project?
To answer additional questions about this project or provide feedback, please contact the program officer.
