|Title:||An Efficacy Study of the MathSpring Personalized Learning System That Responds to Student Affect|
|Principal Investigator:||Feng, Mingyu||Awardee:||WestEd|
|Program:||Education Technology [Program Details]|
|Award Period:||4 years (07/01/19–06/30/23)||Award Amount:||$3,299,446|
Co-Principal Investigators: Schneider, Steven; Woolf, Beverly; Arroyo, Ivon
Purpose: This purpose of this project is to evaluate the efficacy of MathSpring, an online intervention that provides individualized affective and strategic support to students as they solve math problems. The need to improve students' mathematics achievement is well recognized. According to the 2017 National Assessment of Educational Progress, only 34% of American public-school grade 8 students score at or above proficient in mathematics.
Project Activities: MathSpring is a fully developed online intervention that is in widespread use in schools around the country. Results from pilot studies of MathSpring demonstrate its promise for improving student learning in mathematics, and provide a basis for a larger-scale, independent, and rigorous efficacy evaluation.Year 1 will serve as a practice year, during which treatment group teachers will receive professional development to practice using MathSpring to supplement their mathematics instruction. As well in Year 1, the data collection instruments will be piloted, refined, and finalized, and implementation fidelity metrics will be developed. In Year 2 the researchers will implement the intervention in treatment classrooms, and researchers will measure the mathematics achievement of 6th grade students in both treatment and control groups as well as student interest in mathematics. Year 3 will focus on data analysis and dissemination of study findings. Researchers will analyze efficacy using a hierarchical linear regression (HLM) model that examines mean differences in grade 6 achievement between students in treatment and control groups, controlling for prior achievement and other covariates.
Products: The products of this project include evidence of the efficacy of the MathSpring program, conference presentations and peer reviewed publications, and publicly available reports for school administrators.
Setting: The researchers will recruit a representative sample of schools in Massachusetts and ensure the sample includes low-performing schools and schools with lower-socioeconomic status student bodies.
Population/Sample: The study will include 80 mathematics classrooms (sixth grade) with approximately 4,000 students from 40 public schools in the state of Massachusetts.
Intervention: MathSpring is a supplemental online mathematics intervention where students are presented with and solve problems customized to their individual level. MathSpring generates hints if students need assistance. The intervention provides affective support through animated pedagogical agents which share messages designed to promote growth mindset and facilitate reflection to track progress and set goals. A teacher dashboard presents diagnostic reports to facilitate reviews of students' progress, enabling teachers to adapt instruction accordingly. Teachers also receive professional development to support their use of the reports as a formative assessment tool to shape classroom practices.
Control Condition: Teachers assigned to the business-as-usual control condition will continue using existing instructional practices and supplemental technologies (other than MathSpring).
Research Design and Methods: The research team will employ a clustered randomized design to answer a set of research questions about the primary outcome, moderators, mediators, and implementation fidelity. Researchers will access student generated log data from MathSpring. Teachers will be blocked by school and randomly assigned to either the treatment or control condition.Teachers will be asked to participate for two full school years in 2020–21 and in 2021–22. Researchers will collect data on implementation fidelity and contrast between conditions in both years.
Key Measures:The primary measures of student mathematics achievement include the Grade 6 Massachusetts Comprehensive Assessment System (MCAS) assessment and Mathematics Diagnostic Testing Project (MDTP) Grade 7 Readiness Test, and the primary measure for affect is the Attitudes Toward Mathematics Inventory (ATMI).Researchers will assess fidelity in treatment classrooms through researcher developed measures, through observations, and through usage records in MathSpring. Also, are they capturing any information about whether the affective supports are working?
Data Analytic Strategy: The research team will answer the impact question using a three-level hierarchical linear regression model of mean differences in Grade 6 achievement between students in two conditions, controlling for prior achievement and other covariates. They will use moderator analyses to examine the impact of the intervention on the learning of students with low-baseline mathematics achievement, and different demographic backgrounds. In addition, they will conduct mediator analyses to examine the link between student learning outcomes, students' approaches to learning and attitude towards mathematics, and teachers' use of the reports to inform class instruction.
Cost Analysis: Researchers will determine the costs associated with implementing the intervention using the "ingredients method." This will entail systemically collecting and analyzing all expenditures on personnel, facilities, equipment, materials, and training.