Description:

Co-Principal Investigators: Pamela Jordan, Diane Litman

Purpose: Recent studies show that U.S. students lag behind students in other developed countries in math and science. Because one-on-one tutoring has been shown to be a highly effective form of instruction, many educators and education policy makers have looked to intelligent tutoring systems (ITSs) as a means of providing cost-effective, individualized instruction to students that can improve their conceptual understanding of and problem-solving skills in math and science. However, even though many ITSs have been shown to be effective, they are still not as effective as human tutors. The goal of this project is to build an enhanced version of a natural-language dialogue system that engages students in deep-reasoning, reflective dialogues after they solve quantitative problems in Andes, an intelligent web-based tutoring system for physics.

Project Activities: Project activities include using an iterative design process to build, test, and refine a reflective dialogue-enhanced version of Andes. During the third year of the project, the researchers will collect pilot data that addresses the feasibility of implementing the fully developed Andes with Reflective Dialogue System in authentic high school physics classes. Researchers will compare students' pre- and post-test performance on measures of conceptual understanding and problem-solving ability in physics, and compare the performance of students who use the current and enhanced version of the system to those who use the non-dialogue Andes system.

Products: The products of this project will include published reports and a fully developed, enhanced version of Andes for high school physics students.

Structured Abstract

Setting: Research for this project will be conducted in local urban and suburban schools in a large urban city of western Pennsylvania.

Population: Participants for the initial development work will include students enrolled in a first-year physics course at the university, as well as high school students taking physics in the local urban and suburban schools. The pilot studies in Year 3 will take place in two to three authentic high school physics classes in urban and suburban schools.

Intervention: The proposed study will develop an enhanced version of Andes, which will engage students in deep-reasoning, reflective dialogues after they solve quantitative problems. The enhanced system will align its dialogue turns with those of the student by reusing parts of the student's dialogue. The system will adjust the level of abstraction that the student is working from when the student is over-generalizing or missing important concepts; and, it will abstract correct student input when such abstraction will enhance student understanding.

Research Method: Research methods include using an iterative design process to build, test, and refine a reflective dialogue enhanced version of Andes. Researchers will first prepare a prototype for experienced physics teachers and students to try out using the "Wizard of Oz" paradigm—an experiment in which subjects interact with a computer system that subjects believe to be autonomous, but which is actually being operated or partially operated by an unseen human being. Researchers will then identify cases in which the system does not work as intended (for example, the tutor prompts the student to generalize or make distinctions when this is not warranted by the discourse context), refine the software to correct these problems, and test the revised software in a subsequent field trial. The participants during these first two years of studies will include students enrolled in a first-year physics course at an urban university and high school students taking physics in local urban and suburban schools. During the final year of the project, the researchers will collect pilot data that addresses the feasibility of implementing the fully developed Andes with Reflective Dialogue System in authentic high school physics classes. Researchers will compare students' pre- and post-test performance on measures of conceptual understanding and problem-solving ability in physics, and compare the performance of students who use the current and enhanced version of the system to those who use the non-dialogue Andes system.

Control Condition: The researchers will compare the performance of students who use the enhanced version of the system (Andes with Reflective Dialogue) with those who use the current version of Andes (non-dialogue Andes system).

Key Measures: The team will collect three categories of measures: measures of system performance (e.g., computer log data); measures of feasibility of implementing the intervention (e.g., student survey and teacher interviews); and measures of system potential to enhance learning (e.g., pre- and post-tests of students' conceptual understanding and problem-solving ability in physics).

Data Analysis: Researchers will analyze logs collected during each field trial of corrective abstraction and specialization to identify problems and modify the software as necessary. Students' survey ratings on items dealing with system performance will be analyzed to see if they increase across successive "Wizard of Oz" trials and pilot studies. Discussion with teachers will be used to assess the feasibility and usability of the system. Pre- and post-test gains on conceptual versus problem-solving measures will be analyzed to determine growth on learning, and to compare the amount and types of learning across different versions of Andes.

Related IES Projects: Exploring Studies to Derive Policies for Adaptive Natural-language Tutoring in Physics (R305A130441)

Products and Publications

Journal article, monograph, or newsletter

Katz, S., and Albacete, P. (2013). A Tutoring System That Simulates the Highly Interactive Nature of Human Tutoring. Journal of Educational Psychology, 105(4): 1126–1141.

Lipschultz, M., Litman, D., Katz, S., Albacete, P., and Jordan, P. (2014). Predicting Semantic Changes in Abstraction in Tutor Responses to Students. International Journal of Learning Technology, 9(3): 281–303.

Proceeding

Jordan, P., Albacete, P., Ford, M., Katz, S., Lipschultz, M., Litman, D., Silliman, S., and Wilson, C. (2013). The Rimac Tutor—A Simulation of the Highly Interactive Nature of Human Tutorial Dialogue, Interactive Event. In Proceedings of the 9th International Conference of AI in Education, AIED 2013 (pp. 928–929). Memphis, TN: AIED.

Jordan, P., Katz, S., Albacete, P., and Ford, M. (2012). Reformulating Student Contributions in Tutorial Dialogue. In Proceedings of the INLG 2012, 7th International Natural Language Generation Conference (pp. 95–99). Utica, IL: Association for Computational Linguistics.

Katz, S., Albacete, P., Ford, F., Jordan, P., Lipschultz, M., Litman, D., Silliman, S., and Wilson, C. (2013). Pilot Test of a Natural-Language Tutoring System for Physics That Simulates the Highly Interactive Nature of Human Tutoring. In Proceedings of the 16th International Conference of AI in Education (pp. 636–639). Memphis, TN: Springer-Verlag.

Katz, S., Albacete, P., Jordan, P., and Litman, D. (2011). Dialogue Analysis to Inform the Development of a Natural-Language Tutoring System. In Proceedings of the 15th Workshop on the Semantics and Pragmatics of Dialogue (pp. 167–175). Los Angeles: SemDial.

Lipschultz, M., Litman, D., Jordan, P., and Katz, S. (2012). Evaluating Learning Factors Analysis. In Proceedings of the UMAP'12 20th International Conference on User Modeling, Adaptation, and Personalization. Montreal, Canada: Springer-Verlag.