|Efficacy of Zearn Math
|Pane, John F.
|Science, Technology, Engineering, and Mathematics (STEM) Education [Program Details]
|3 years (10/01/2022 – 09/30/2025)
Co-Principal Investigators: Doss, Christopher; Tekkumru-Kisa, Miray
Purpose: Zearn Math has shown promise for improving student mathematics achievement, but its efficacy has not yet been established through a rigorous, well-powered evaluation. Researchers at the non-profit RAND Corporation (RAND) will evaluate the efficacy of Zearn Math for improving student mathematics outcomes in grades 3 to 5. The study will employ a randomized controlled trial, complemented by studies of implementation and cost. RAND will conduct this evaluation independently of Zearn, the non-profit organization that developed and disseminates Zearn Math.
Project Activities: The research team will conduct a comprehensive set of analyses to characterize the effect of Zearn Math on student outcomes. The three facets of this study include (1) an outcomes study that leverages a randomized controlled trial design to address confirmatory and exploratory questions; (2) an implementation study that examines fidelity, service contrast, and factors that affect implementation; and (3) a cost and cost-effectiveness study.
Products: The research team will produce peer-reviewed journal articles and accessible derivative products such as research briefs, blog posts, media coverage, and conference presentations. Members of the research team will develop a suite of materials that can support presentation of key findings to audiences of educators and policymakers as well as researchers and will leverage a range of partnerships and venues to disseminate study findings.
Setting: The sample for this study will consist of students in grades 3 to 5 in elementary schools serving all of those grades in a large urban Texas school district. The district has 274 schools — 160 of which are elementary schools — and more than 194,000 students.
Sample: The study will recruit 64 schools into the study, resulting in a sample of an estimated 768 classrooms and 15,360 students in grades 3 to 5 each year. Less than 10 percent of students in the district are White, 22 percent are Black, 62 percent are of Hispanic descent, and 79 percent are from economically disadvantaged families.
Intervention: The intervention, Zearn Math, will be used as a digital complement to the district's existing mathematics curriculum. Zearn Math is built on research-based principles of high-quality instructional materials and a recognition that such materials are most impactful when well-supported by intensive, curriculum-specific professional learning opportunities. Zearn's approach uses engaging independent digital lessons to differentiate grade-level instruction for all students, designed to integrate with the existing mathematics curriculum to create a cohesive classroom system. Educators are supported with professional development (PD) based on pedagogical content knowledge and actionable reports on student learning. Zearn Math focuses deeply on a few interconnected math concepts, helping students build a strong mathematical foundation that lets them apply concepts flexibly to a variety of situations, contexts, and problems. As students work through the digital lessons aligned to core grade level math instruction, they learn and practice new concepts at their own pace with concrete and digital manipulatives, interactive videos, pictorial representations, paper-and-pencil transfer, and precise digital feedback when they reveal a misconception. The software is designed to accommodate learning differences and foster positive math mindsets and social belonging, aspiring for all students to love learning math.
Research Design and Methods: The research team will use a blocked, cluster-randomized design to balance the feasibility of strong implementation with maximizing statistical power and minimizing other threats such as spillover or attrition. The researchers will randomize schools within blocks of similar schools based on factors such as poverty, achievement, or geographic locale. Within each block, the research team will randomize half of the schools to the treatment group that implements Zearn Math and the other half to a business-as-usual control group. In the implementation study, researchers will examine the extent to which Zearn Math is implemented as intended, and they will also compare mathematics instruction in both the treatment and control groups. Lastly, the cost study will examine both costs and cost-effectiveness of Zearn Math.
Control Condition: The control condition will continue with business-as-usual mathematics instruction.
Key Measures: The primary student outcome for confirmatory analyses is student performance on state grade-level mathematics standards, as measured by the Texas STAAR assessment. Exploratory analyses consider student growth in mathematics achievement measured by the adaptive online Renaissance Star Mathematics assessment. For the implementation study, the research team will use a range of qualitative and quantitative data, including an annual teacher survey, a teacher log, and annual case studies in 4 to 5 schools.
Data Analytic Strategy: The research team will use confirmatory outcomes analyses to investigate the cumulative effects of Zearn Math over 2 academic years on students' mathematics grade-level proficiency on the Texas state assessment—overall and for the subgroup of students who enter the study scoring below grade-level proficiency. Exploratory analyses will examine 1-year effects and cumulative effects for various other subgroups and effects on mathematics achievement growth. Lastly, the researchers will explore how metrics like use of the Zearn software and fidelity of implementation are related to student outcomes. These outcomes analyses will be complemented by studies of the implementation and cost of Zearn Math.
Cost analysis: The cost analysis will identify all the ingredients and identify both recurring and non-recurring cost aspects. Because the Zearn Math program continues for multiple years, costs are amortized to arrive at an estimate associated with a single year. The cost-effectiveness analysis extends the cost analysis by combining two-year per-student costs with measures of the intervention effect over two years on mathematics achievement.