Project Activities
This intervention included (1) a web-based platform that provides support to students as they solve mathematics problems from the district-assigned textbooks and provides detailed student-level and class-level formative assessment data to teachers to help inform adjustments in classroom instruction and pacing and (2) professional development to support teacher use of the reports as a formative assessment tool. Schools were randomly assigned to either the treatment or control condition and participated for 2 full school years. Data on implementation fidelity and contrast between conditions were collected in both years.
Structured Abstract
Setting
Participating schools were located in rural, suburban, and urban districts across North Carolina. The sample included low-performing schools and students from lower-socioeconomic status backgrounds.
Sample
The project originally recruited 63 schools from 41 different districts into the sample and randomly assigned to condition (32 treatment schools and 31 control schools). The schools were distributed across rural, town, suburban, and city communities (33 rural, 11 town, 8 suburban, and 11 city). Of the 63 schools, 18 were charter schools, 45 were public schools, and 48 of the schools received Title I funding. The original student sample included 4,958 students (2964 treatment and 2264 control).
Intervention
ASSISTments is a web-based platform that provides support to students as they solve mathematics problems pulled from district-assigned textbooks and teacher-created worksheets and that provides detailed student-level and class-level formative assessment data to teachers to help inform adjustments in classroom instruction and pacing. ASSISTments also provides teachers with access to pre-organized problem sets, called Skill Builders, which teachers can assign to students. Skill Builders provide students with additional practice, hints, and guidance as they work through problems and immediate feedback to answers.
Research design and methods
The study used a school-level, delayed-treatment, clustered randomized experimental design. Schools were randomly assigned to either the treatment or control condition and all teachers in grade 7 participated in the same condition. Schools were asked to participate for 2 school years (2018-19 and 2019-20). Data on implementation fidelity and contrast between conditions were collected in both years. In the treatment schools, teachers and students in grade 7 used the ASSISTments to support their homework assignment and review practices. Teachers assigned homework online and receive diagnostic reports to facilitate their review of homework and adapt their instruction accordingly. Students completed their homework on their computers and received (a) immediate feedback on their answers to textbook problems, (b) hint messages to help solve difficult problems, (c) Skill Builders that track and adjust to their mastery status of knowledge, and (d) automatic reassessment to help improve their retention of previous mastered skills. Teachers received professional development and ongoing support to support their use of the reports as a formative assessment tool. A local coach conducted on average three classroom visits in each school year.
Control condition
In the control condition, teachers continued with their existing instructional practices and technologies (other than ASSISTments). Teachers who did not have a formal homework policy were recommended to assign a minimum of 20 minutes of homework per night for at least 3 nights a week.
Key measures
The North Carolina End-of-Grade assessment (EoG) was originally the primary measure of student math achievement. Due to COVID-19, the EoG test was canceled in Spring 2020, and the team administered an online the grade 8 math readiness test as a supplemental measure. Teacher logs were used to measure teacher homework review practices. Other measures include eligibility for free or reduced-price lunch as a measure of socioeconomic status (SES), school size, rural or non-rural status of school, individualized education plan status, and English proficiency status.
Data analytic strategy
The team used a three-level hierarchical linear regression model (HLM) to examine mean differences in scores of the grade 8 math readiness test between students in two conditions, controlling for prior achievement and other covariates. Researchers also completed moderator analyses, examining whether ASSISTments had differential impact on students with different characteristics. The student characteristics investigated included gender, grade 6 math achievement, disability status, and economically disadvantaged status. Two-level HLM analyses were conducted to examine the effect of the program on teacher practice in performing targeted homework review. Finally, researchers conducted exploratory analyses to examine the link between student learning outcomes and student and teacher use of the ASSISTments, using a three-level hierarchical linear regression model with student, classroom/teacher, and school levels.
Key outcomes
Key outcomes will be included once findings from the replication study are published.
People and institutions involved
IES program contact(s)
Products and publications
ERIC Citations: Find available citations in ERIC for this award here and here.
Publicly Available Data: Study data have been deposited in the Open ICPSR (https://www.openicpsr.org/openicpsr/project/183645/version/V1/view), including participant research ID, randomization condition, ASSISTments usage data, teacher log/survey data, and student MDTP assessment data. Data-sharing agreements between researchers and the state did not allow state administrative data including school enrollment, student demographic, and student state test data to be shared with third parties.
Additional Online Resources and Information: WPI and the ASSISTments Foundation (TAF) hosted webinars for teachers and created a library of recorded webinars (https://new.assistments.org/webinar-library). The project produced blogs (https://new.assistments.org/blog). New intervention materials were posted on the developer’s website (e.g., ASSISTments Advantage (https://new.assistments.org/assistments-advantage)).
Select Publications:
Proceedings
Baral, S., Botelho, A., Erickson, J.A., Benachamardi, P., & Heffernan, N. (2021). Improving Automated Scoring of Student Open Responses in Mathematics. In Hsiao, Sahebi, Bouchet & Vie (eds). Proceedings of the 14th International Conference on Educational Data Mining (EDM2021). pp 130-138.
Gillespie, J., Winn, K., Faber, M. & Hunt, J. (2022). Implementation of a Mathematics Formative Assessment Online Tool Before and During Remote Learning. In Proceedings of the 23rd International AI in Education (AIED) Conference. pp 168-173. Duram, UK. July 2022.
Patikorn, T. & Heffernan, N. T. (2020, August 12) Effectiveness of Crowd-Sourcing On-Demand Tutoring from Teachers in Online Learning Platforms. Proceedings of the Seventh ACM Conference on Learning @ Scale (L@S). pp 115-124. doi.org/10.1145/3386527.3405912.
Prihar, E., Haim, A., Sales, A., & Heffernan, N. (2022). Automatic Interpretable Personalized Learning. Proceedings of the Ninth ACM Conference on Learning @ Scale (L@S ’22), June 1-3, 2022, New York City, NY, USA. doi.org/10.1145/3491140.3528267
Prihar, E., Patikorn, T., Botelho, A., Sales, A., & Heffernan, N. (2021). Toward Personalizing Students' Education with Crowdsourced Tutoring. Learning@Scale 2021. pp 37-45 doi.org/10.1145/3430895.3460130
Prihar, E., Syed, M., Ostrow, K., Shaw, S., Sales, A., & Heffernan, N. (2022). Exploring Common Trends in Online Educational Experiments. In Proceedings of the 15th International Educational Data Mining Conference, Durham, England.
Shen, J.T., Yamashita, M., Prihar, E., Heffernan, N., Wu, X., McGrew, S., & Lee, D. (2021). Classifying Math Knowledge Components via Task-Adaptive Pre-Trained BERT. 22nd International Conference on Artificial Intelligence in Education. pp 408- 419. doi.org/10.1007/978-3-030-78292-4_33
Zhang, M., Baral, S., Heffernan, N. & Lan, A. (2022). Automatic Short Math Answer Grading via In-context Meta-learning. In Proceedings of the 15th International Conference on Educational Data Mining (EDM2022).
Additional project information
Previous award details:
Previous award number:
R305A170137
Related projects
Supplemental information
Co-Principal Investigators: Heffernan, Neil; Roschelle, Jeremy; Murphy, Robert
Questions about this project?
To answer additional questions about this project or provide feedback, please contact the program officer.
