Co-Principal Investigator: Sorensen, Nicholas

Purpose: This purpose of this project is to evaluate the effectiveness of a promising web-based adaptive mathematics tutoring and homework program. The study will contribute to an area where there is limited rigorous evidence in support of online programs that improve student outcomes in science, technology, engineering, and mathematics.

Project Activities: ASSISTments is a fully-developed web-based mathematics program that formatively assesses and then supports student learning as homework problems are completed. The program also provides teachers with reports on student performance to guide instruction. Prior research demonstrates the efficacy of ASSISTments to improve students' mathematics learning under researcher controlled conditions. In this study, ASSISTments will be implemented and evaluated under routine conditions. In Year 1, researchers will carry out recruitment and random assignment activities. In Years 2 and 3, treatment schools will implement ASSISTments with their seventh grade students multiple times per week.

Products: The products of this project include evidence of the effectiveness of the ASSISTments program, and peer reviewed publications.

Structured Abstract

Setting: Eighty rural and urban schools from seven geographically, culturally, and economically diverse sites across six states (New York, Massachusetts, Illinois, California, Texas, and Wisconsin) will participate in the study.

Population/Sample: The study will include 96 seventh-grade teachers and 16,000 seventh-grade students—at least 25 percent of whom will be eligible for free or reduced-priced lunch.

Intervention: ASSISTments is a web-based program that provides immediate feedback to students and teachers on homework, and it can be used with any commercial or locally developed math curriculum. While doing homework in ASSISTments, students receive support including immediate feedback on the correctness of their answers and tutoring. Further, teachers can assign "mastery" problem sets that organize practice to facilitate the achievement of proficiency.

Control Condition: Seventh-grade classrooms within the study's control schools will follow typical homework routines without the support of ASSISTments during the two-year evaluation window.

Research Design and Methods: The design is a multisite, cluster randomized controlled trial. Schools will be block-randomly assigned to condition within strata and site. Treatment schools will implement ASSISTments for two years with two cohorts of seventh-grade students (2018–19 and 2019–20). Student outcomes will be collected for both cohorts.

Key Measures: Primary outcome measures include seventh-grade state mathematics assessment scores from spring 2019 (Cohort 1) and spring 2020 (Cohort 2), using the relevant accountability measures of math achievement within each state. The research team will use an algebra readiness assessment, student surveys of math mindset, teacher surveys, and video observations of teacher homework review practices for exploratory research questions. They will use participation in training, teacher and student usage data from the ASSISTments system, administrator surveys, and ASSISTments financial data to examine fidelity and cost of implementation.

Data Analytic Strategy: Impact analyses will follow an intent-to-treat approach. To estimate the treatment effect of ASSISTments at the cluster (school) level based on the level at which treatment was randomly assigned, researchers will employ a multilevel model on a nested data structure of students within schools and within blocks. The research team will use exploratory meditational analyses to employ multilevel structural equation modeling.

Related IES Projects:
Making Longitudinal Web-Based Assessments Give Cognitively Diagnostic Reports to Teachers, Parents, and Students While Employing Mastery Learning
An Efficacy Study of Online Mathematics Homework Support: An Evaluation of the ASSISTments Formative Assessment and Tutoring Platform
Efficacy of ASSISTments Online Homework Support for Middle School Mathematics Learning: A Replication Study