Description:

Purpose: Helping students achieve proficiency in mathematics, as measured by state standardized tests, is a priority for mathematics teachers. To help meet this goal, teachers need to know how well their students are performing throughout the school year, so that instruction can be tailored to better fit the needs of students. In addition, students and parents should be able to keep track of mathematical concepts or skills that students have not mastered to know which areas need extra practice and attention. The purpose of this project is to develop a computer-based assessment and tutoring system designed to track and support mastery learning in mathematics among sixth- and seventh-graders, and to conduct initial evaluations of the system.

Project Activities: The researchers are developing and refining components to the ASSISTments system and conducting an initial evaluation to assess the potential efficacy of the ASSISTments system on mathematics learning. The ASSISTments system is a web-based assessment program that provides tutoring on mathematics questions that students get wrong. The system offers tutoring associated with each of the 300 released eighth-grade Massachusetts state mathematics assessment items. The researchers are developing a mastery learning tracking component to the program so that teachers receive instantaneous feedback on their students' progress with multiple types of web-based reports, including individual tracking for 98 different skills. In addition, the researchers are developing an automated component to the ASSISTments system that will deliver weekly reports to parents detailing what their children have learned, as well as what specific skills they are struggling with.

Products: The products from this project include a web-based tutoring system in mathematics for sixth- and seventh-grade students and teachers, and published reports on the initial evaluations of the potential efficacy of the ASSISTments system on student learning.

Structured Abstract

Purpose: The purpose of this project is to develop a computer-based assessment and tutoring system designed to track and support mastery learning in mathematics among sixth- and seventh-graders, and to conduct initial evaluations of the system.

Setting: The schools are located in Massachusetts.

Population: Participants are sixth- and seventh-grade math students and their teachers from an urban school district with a high percentage of ethnic minority and low socioeconomic status students.

Intervention: The ASSISTments system is a web-based assessment program that provides tutoring on mathematics questions that students get wrong. The system offers tutoring associated with each of the 300 released eighth-grade Massachusetts state mathematics assessment items. The researchers are developing a mastery learning tracking component to the program so that teachers receive instantaneous feedback on their students' progress with multiple types of web-based reports, including individual tracking for 98 different skills. In addition, the researchers are developing an automated component to the ASSISTments system that will deliver weekly reports to parents detailing what their children have learned, as well as what specific skills they are struggling with.

Research Design and Methods: In addition to developing and refining the components of the ASSISTments system, the researchers are performing a series of experiments to investigate the potential efficacy of the ASSISTments system on student achievement in mathematics. In the first study, eight seventh-grade teachers will have two of their classes randomly assigned to use the ASSISTments system and two of their classes to the control condition (i.e., each teacher serves as his/her own control). In addition, two sixth-grade teachers will be randomly assigned to use the ASSISTments system and two to the control condition In a second study, the researchers will examine the potential effect of increased communication to parents (through email, phone messages, and print messages) on student learning by randomly assigning classes to the experimental condition (ASSISTments system with parent notification of student progress) and the control condition (ASSISTments system only). The same set of teachers from the first study will be recruited to participate in the second study. The outcomes of these studies will be used to determine the final components of the revised system. In the final two years of the project, the researchers will evaluate the potential efficacy of the revised ASSISTments system. Elementary schools will be matched and randomly assigned to implement the ASSISTments system. At least one sixth-grade class from 16-20 elementary schools will participate.

Control Condition: Depending on the intervention, classrooms in the control condition either do not use the ASSISTments system at all, or use it without the parental notification component.

Key Measures: The main measure of interest is gain scores on the Massachusetts Comprehensive Assessment System between the end of sixth grade and the end of seventh grade.

Data Analytic Strategy: This development project is intended only to obtain evidence of the potential efficacy of the intervention. For the first two studies, analyses will be conducted at the classroom level, but in the final summative evaluation, analyses will be conducted at the school level.

Project Website: https://www.assistments.org/

Related IES Projects: Using Web-based Cognitive Assessment Systems for Predicting Student Performance on State Exams (R305K030140) and An Efficacy Study of Online Mathematics Homework Support: An Evaluation of the ASSISTments Formative Assessment and Tutoring Platform (R305A120125)

Products and Publications

Book chapter

Baker, R., Pardos, Z., Gowda, S., Nooraei, B., and Heffernan, N. (2011). Ensembling Predictions of Student Knowledge Within Intelligent Tutoring Systems. In J.A. Konstan, R. Conejo, J.L. Marzo, and N. Oliver (Eds.), User Modeling, Adaption and Personalization, Volume 6787 (pp. 13–24). Heidelberg: Springer.

Heffernan, N., Heffernan, C., Decoteau, M., and Militello, M. (2012). Effective and Meaningful Use of Educational Technology: Three Cases From the Classroom. In C. Dede, and J. Richards (Eds.), Digital Teaching Platforms (pp. 88–102). New York: Teacher's College Press.

Singh, R., Saleem, M., Pradhan, P., Heffernan, C., Heffernan, N., Razzaq, L. and Dailey, M. (2011). Feedback During Web-Based Homework: The Role of Hints. In G. Biswas, S. Bull, J. Kay, and A. Mitrovic (Eds.), Artificial Intelligence in Education: 15th International Conference, AIED 2011, Volume 6738 (pp. 328–336). Heidelberg: Springer.

Journal article, monograph, or newsletter

Baker, R.D., Goldstein, A.B., and Heffernan, N.T. (2011). Detecting Learning Moment-By-Moment. International Journal of Artificial Intelligence in Education, 21(1): 5–25.

Broderick, Z., DeNolf, K., Dufault, J., Heffernan, N., and Heffernan, C. (2011). Increasing Parent Engagement in Student Learning Using an Intelligent Tutoring System. Journal of Interactive Learning Research, 22(4): 523–550.

Feng, M., Heffernan, N., and Koedinger, K. (2009). Addressing the Assessment Challenge With an Online System That Tutors as it Assesses: User Modeling and User-Adapted Interaction. The Journal of Personalization Research, 19(3): 243–266.

Gong, Y., Beck, J.E., and Heffernan, N.T. (2011). How to Construct More Accurate Student Models: Comparing and Optimizing Knowledge Tracing and Performance Factor Analysis. International Journal of Artificial Intelligence in Education, 21(1): 27–46.

Mendicino, M., Razzaq, L., and Heffernan, N.T. (2009). A Comparison of Traditional Homework to Computer-Supported Homework. Journal of Research on Technology in Education, 41(3): 331–359.

Ostrow, K.S., Heffernan, N.T., and Williams, J.J. (2017). Tomorrow's EdTech Today: Establishing a Learning Platform as a Collaborative Research Tool for Sound Science. Teachers College Record,, 119(3): 1–36.

Proceeding

Bahador, N., Pardos, Z., Heffernan, and Baker, R. (2011). Less is More: Improving the Speed and Prediction Power of Knowledge Tracing by Using Less Data. In Proceedings of the 4th International Conference on Educational Data Mining (pp. 101–110). Eindhoven, Netherlands: Educational Data Mining.

Feng, M., Beck, J., Heffernan, N., and Koedinger, K. (2008). Can an Intelligent Tutoring System Predict Math Proficiency as Well as a Standardized Test?. In Proceedings of the 1st International Conference on Education Data Mining (pp. 107–116). Montreal, Canada: Educational Data Mining.

Pardos, Z., Gowda, S., Baker, R., and Heffernan, N. (2011). Ensembling Predictions of Student Post-Test Scores for an Intelligent Tutoring System. In Proceedings of the 4th International Conference on Educational Data Mining (pp. 189–198). Eindhoven, Netherlands: Education Data Mining.

Qiu, Y., Qi, Y., Lu, H., Pardos, Z., and Heffernan, N. (2011). Does Time Matter? Modeling the Effect of Time With Bayesian Knowledge Tracing. In Proceedings of the 4th International Conference on Educational Data Mining (pp. 139–148). Eindhoven, Netherlands: Educational Data Mining.

Razzaq, L., and Heffernan, N. (2009). To Tutor or Not to Tutor: That is the Question. In Proceedings of the 2009 Artificial Intelligence in Education Conference (pp. 457–464). Brighton, UK: IOS Press.

Singh, R., Saleem, M., Pradhan, P., Heffernan, C., Heffernan, N., Razzaq, L. Dailey, M. O'Connor, C., and Mulchay, C. (2011). Feedback During Web-Based Homework: The Role of Hints. In Proceedings of the Artificial Intelligence in Education Conference 2011 (pp. 328–336). Heidelberg, Germany: Springer.

Trivedi, S., Pardos, Z., Sarkozy, G., and Heffernan, N. (2011). Spectral Clustering in Educational Data Mining. In Proceedings of the 4th International Conference on Educational Data Mining (pp. 129–138). Eindhoven, Netherlands: Educational Data Mining.