Project Activities
In this study, researchers tested the efficacy of the intervention using a randomized experimental design with two cohorts of teachers. Teachers spent their first year in the study implementing the intervention and learning how to use it. During the second year of the intervention, student performance data were used to evaluate the efficacy of the intervention. Each teacher participated in the study for two years. Teachers received professional development training and support on the use of ASSISTments both prior to and throughout the school year, where they used the system for homework at least four times per week. The technology in ASSISTments enabled teachers to assign customized homework to their students, aligned to individual student needs. Teachers also assigned "mastery" problem sets that organized practice to facilitate the achievement of proficiency. Students completed their homework using the system. While doing homework in ASSISTments, students received support including immediate feedback on the correctness of their answers and extensive tutoring.
Structured Abstract
Setting
The study was conducted in public middle/junior high schools in the state of Maine.
Sample
The sample for this study included 43 public middle/junior high schools with 2,850 grade 7 students. The schools were recruited in 2 cohorts with a 1-year delay in-between to allow the project team time to recruit. The majority of students in the sample were White (92.6 percent), and many qualified for free and reduced-price lunch (38.7 percent). Students with individualized education programs (IEPs), an indicator of the need for special education services, made up about 12 percent of the sample.
Teachers and students in grade 7 used the ASSISTments, developed in IES grants from 2003 and 2007, to support their nightly homework. Teachers assigned nightly homework online and receive cognitive diagnostic reports to facilitate their review of individual student homework and adapted their instruction accordingly. In Maine, all middle school students have individual laptops. Students completed their homework on their laptop computers and received (1) immediate feedback on their answer, (2) individualized tutoring and hint messages on difficult problems, (3) mastery problem sets that adjusted to their mastery status of knowledge, and (4) automatic reassessment of a subset of skills to help improve their retention of previously mastered skills. Teachers received intensive professional development to support them using the reports as a formative assessment tool and parents received reports about their children's progress and homework performance.
Research design and methods
The study used a randomized experimental design, with participation occurring over 2 years for each teacher. In the first year, teachers implemented and become familiar with the intervention. Only the second year's usage of ASSISTments were used in evaluating the effects of the intervention on student performance. There were two cohorts of teachers, with cohort 1 participating over years 1 and 2 and cohort 2 over years 2 and 3. The unit of random assignment was schools. A total of 43 schools were randomly assigned to either the treatment or control condition.
Control condition
Teachers in schools assigned to the control condition continued to use the instructional practices they were currently using or that were made available to them during the course of the study, including all traditional and formative assessment practices, other than ASSISTments.
Key measures
Statewide math test scores from the New England Common Assessment (NECAP) along with a nationally normed mathematics achievement test, TerraNova, were the primary outcome measures used in the study. Additionally, researcher-developed pre- and posttests were used for two focal units in grade 7 curriculum to obtain proximal measures of achievement.
Data analytic strategy
A three-level hierarchical linear regression model (students nested within teachers within schools) was used to account for the effect of clustering on the variance structure of the data. When cohort 2 completed participation in the study (i.e., at the end of year 3), the data from both cohorts were combined and analyzed. Moderator analyses examined the differential impact of the intervention for subgroups of students, including learners with low-baseline math achievement, students with IEPs, English language learners, and students who qualify for free and reduced-price lunch. Finally, mediation analyses examined the link between teachers' use of the ASSISTments and student homework completion rate.
Key outcomes
The main findings for this project are as follows:
- The ASSISTments intervention significantly increased student scores on an end-of the-year standardized mathematics assessment as compared with a control group that continued with existing homework practices (Roschelle, Feng, Murphy, and Mason, 2016).
- The effect of the ASSISTments intervention was greater for lower-performing students than for higher-performing students (Roschelle, Feng, Murphy, and Mason, 2016).
People and institutions involved
IES program contact(s)
Project contributors
Products and publications
ERIC Citations: Find available citations in ERIC for this award here.
WWC Review: Roschelle, J. Feng, M. Murphy, R.F., and Mason, C.A. (2016). Online Mathematics Homework Increases Student Achievement. AERA Open, 2(4).
Project Website: https://new.assistments.org/
Select Publications:
Journal articles
Heffernan, N.T., Ostrow, K.S., Kelly, K., Selent, D., Van Inwegen, E.G., Xiong, X., and Williams, J.J. (2016). The Future of Adaptive Learning: Does the Crowd Hold the Key? International Journal of Artificial Intelligence in Education, 26(2): 615-644.
Murphy, R., Roschelle, J., Feng, M. and Mason, C.A. (2020) Investigating Efficacy, Moderators and Mediators for an Online Mathematics Homework Intervention, Journal of Research on Educational Effectiveness, 13(2), 235-270.
Ostrow, K. S., Wang, Y., and Heffernan, N. T. (2017). How Flexible is Your Data? A Comparative Analysis of Scoring Methodologies Across Learning Platforms in the Context of Group Differentiation. Journal of Learning Analytics, 4(2): 91-112.
Proceedings
Adjei, S.A., Botelho, A.F., and Heffernan, N.T. (2016). Predicting Student Performance on Post-Requisite Skills Using Prerequisite Skill Data: An Alternative Method for Refining Prerequisite Skill Structures. In Proceedings of the Sixth International Conference on Learning Analytics and Knowledge (pp. 469-473). New York: ACM.
Feng, M (2014). Towards uncovering the mysterious world of math homework. In Proceedings of the 7th International Conference on Educational Data Mining (pp. 425-426).
Feng, M., Roschelle, J., Bhanot, R. and Mason, C (2016). Investigating gender differences on homework in middle school mathematics. In Proceedings of the 9th International Conference on Educational Data Mining (pp. 364-369).
Feng, M., Roschelle, J., Heffernan, N., Fairman, J., and Murphy, R. (2014). Implementation of an intelligent tutoring system for online homework support in an efficacy trial. In Proceedings of the 12th International Conference on Intelligent Tutoring Systems, (pp. 561-566).
Feng, M., Roschelle, R., Murphy, R. and Heffernan, N. (2014). Using analytics for improving implementation fidelity in a large scale efficacy trial. In Learning and becoming in practice: The International Conference of the Learning Sciences (ICLS) (pp. 527-534).
Kehrer, P., Kelly, K.M., and Heffernan, N.T. (2013). Does Immediate Feedback While Doing Homework Improve Learning? In Proceedings of the Twenty-Sixth International Florida Artificial Intelligence Research Society Conference (pp. 542-545).
Project website:
Related projects
Questions about this project?
To answer additional questions about this project or provide feedback, please contact the program officer.
