Purpose: In this project, the research team will continue to develop and test Rimac, a tutorial dialogue system for physics developed with prior IES funding (Improving a Natural-Language Tutoring System that Engages Students in Deep Reasoning Dialogues about Physics) and currently serving as the test bed for a current exploration project (Exploring Studies to Derive Policies for Adaptive Natural-Language Tutoring in Physics). The researchers will create and incorporate a student modeling engine into the system and will then conduct a pilot study to assess the degree to which student modeling improves the effectiveness of Rimac. The findings from this research will have broader implications for developing tutoring systems in other subject areas, building domain-neutral tutor authoring environments, and determining the right balance between helping students and encouraging autonomy. Even though several tutorial dialogue systems (i.e., systems that emulate a human tutor by conversing with students via typed or spoken natural language) have been shown to be nearly as effective as human tutors, these systems usually fail to outperform systems that provide students with canned text as feedback instead of dialogue. One possible reason why tutorial dialogue systems have not yet reached their full potential is that they lack the ability to track a student's level of mastery about particular curriculum elements (e.g., concepts, skills) during tutoring due to their lack of a student modeling engine. A student modeling engine would allow the tutor to assess student ability and determine the right level of support as students engage in conversations with the automated tutor.

Project Activities: In Years 1 and 2, researchers will create the student modeling engine and incorporate it into Rimac using an iterative cycle which will consist of alternating phases of software development, field testing, and refinement. In Year 3, researchers will conduct a final field test followed by the pilot study. The pilot study will compare Rimac with the student modeling engine to Rimac without the student modeling engine.

Products: The products of this project include an enhanced version of Rimac, a tutorial dialogue system for physics intended for high school physics students, and peer-reviewed publications.

Structured Abstract

Setting: Participating high schools will be located in urban and suburban areas in Pennsylvania. The participating university will be located in an urban area in Pennsylvania.

Sample: Approximately 80 undergraduate students who have taken physics in high school but not in college and 120 high school students currently enrolled in physics will participate in field trials (about 20 undergraduate students and 30 high school students per field trial). Approximately 160 high school students enrolled in physics will participate in the pilot study. Researchers will include students from a range of high schools to ensure a socioeconomically diverse sample.

Intervention: Rimac is a tutorial dialogue system that presents students with conceptual questions about just-solved physics problems and provides support to students as they answer those questions. The enhanced version of Rimac, which will be developed in this project, will include a student modeling engine. This engine will support the generation of adaptive tutorial dialogues by assessing a student's ability and determining the right level of support to provide to students.

Research Design and Methods: In Years 1 and 2, researchers will create the student modeling engine and incorporate it into Rimac using an iterative cycle which will consist of alternating phases of software development, field testing, and refinement. One field trial will compare alternative approaches to dialogue selection and student model updating approaches. Students will take a pre-test to initialize the student model and then will interact with the system. To determine the best dialogue selection and student model updating approaches, system output will be compared to data collected from human judges on the same tasks (dialogue selection and student model updating). Human judges will be provided with the same information the system receives about each student and will be asked to choose the dialogues most appropriate for the student (dialogue selection) and to rate whether students received the appropriate level of support (model updating). The system approach that most closely matches the human judges' response will be deemed the ‘best' approach. Researchers will run four additional field trials to refine the system design. Students will take a pre-test, interact with the system, and then take a post-test. In Year 3, researchers will conduct a randomized controlled trial. The team will randomly assign students to a group that uses Rimac with the student modeling engine or to a group that uses Rimac without the student modeling engine. All students will take a pre-test, interact with the system, and then take a post-test. The study will take place over two to four sessions depending on classroom schedules. Students will fill out a "user satisfaction" survey after completing the study.

Control Condition: In the control condition, students will interact with a version of Rimac that does not include the student modeling engine. They will solve the same set of problems in the system, view problems in the same order, and engage in the same set of reflective dialogues as students in the intervention condition. In this version of Rimac, dialogues will not adapt to what students know or do not know.

Key Measures: Primary measures will include researcher-developed pre- and post-tests, which will draw upon items from a standard instrument of physics conceptual understanding, the Force Concept Inventory. Researchers will measure prior knowledge using students' Preliminary Scholastic Aptitude Test score. The research team will use percent of correct responses during reflective dialogues to see how well the tutor kept the discussion at the right difficulty level. Researchers will use log files to track student engagement and fidelity of implementation. The "user satisfaction" survey administered to students will include questions about the usability of the system.

Data Analytic Strategy: The research team will perform a one-way analysis of variance for each outcome variable of interest to compare the two versions of the Rimac system. The research team will use multiple regression to study the possible moderating role of student prior knowledge and student aptitude.

Related IES Projects: Improving a Natural-Language Tutoring System that Engages Students in Deep Reasoning Dialogues about Physics (R305A100163), Exploring Studies to Derive Policies for Adaptive Natural-Language Tutoring in Physics (R305A130441)

Project Website: https://sites.google.com/site/rimacsite/home