Grant Closed

DeepTutor: An Intelligent Tutoring System Based on Deep Language and Discourse Processing and Advanced Tutoring Strategies

NCER
Program: Education Research Grants
Program topic(s): Education Technology
Award amount: $1,650,272
Principal investigator: Vasile Rus
Awardee:
University of Memphis
Year: 2010
Project type:
Development and Innovation
Award number: R305A100875

Purpose

Encouraged by the effectiveness of one-on-one human tutoring, computer tutors that mimic human tutors have been built successfully with the hope that a computer tutor could be afforded by every child with access to a computer. The development of state-of-the-art computer tutors with natural language dialogue has been inspired primarily from studies of typical, non-expert human tutors. The proposed project will develop and test an innovative intelligent tutoring system (ITS) that intends to improve the effectiveness of state-of-the-art tutoring systems with natural language dialogue by addressing illusions of tutoring. Researchers focusing on the tutor-tutee relationship have identified a number of illusions that occur during the tutoring process, and those affecting the meta-cognitive and meta-communicative knowledge of tutors and tutees have been identified to include: illusion of grounding, illusion of feedback accuracy, illusion of discourse alignment, illusion of student mastery, and illusion of knowledge transfer. These illusions are thought to result in the tutoring process being less efficient for a number of reasons, such as: the tutor believes the tutee understands more than he really does; the tutor provides feedback that is more concerned with remaining polite than providing constructive criticism; and the student does not fully understand what tutor is saying (e.g., understand the hints). The goal of this project is to address these illusions in order to improve tutoring quality and learning. This research team believes that tutoring quality and learning will improve by penetrating these five illusions of tutoring. Researchers will develop and evaluate an ITS which will address these illusions, and which will integrate recent advances in instructional and curriculum design. Called DeepTutor, the system is intended to improve student outcomes in science relative to a current state-of-the art tutoring system, called AutoTutor, and standard classroom instruction.

Project Activities

The evaluation will focus primarily on learning gains (curriculum-based post-test scores are significantly higher than pre-test scores) as well as on usability and students' perception of the system. The effects of DeepTutor will be compared to a state-of-the-art tutoring system, AutoTutor, and to classroom instruction. Students in the two tutoring conditions, DeepTutor and AutoTutor, will undergo prerequisite mastery and pre-training periods to build needed some prior knowledge before solving problems with the tutors.

Structured Abstract

Setting

This study will take place in Shelby County Schools, a suburban school district surrounding Memphis, TN.

Sample

Participants for this study include high school students in conceptual physics.
Intervention
DeepTutor is a dialog-based tutoring system focused on high school physics which is intended to increase the effectiveness of existing intelligent tutoring systems that hold a conversation in natural language. In particular, researchers will address the five frequent illusions mentioned above—illusion of grounding, illusion of feedback accuracy, illusion of discourse alignment, illusion of student mastery, and illusion of knowledge transfer—through a combination of deep language and discourse processing, recent advances in instructional and curriculum design, and advanced tutoring strategies. Based on technological advances, DeepTutor is expected to provide accurate assessment, better communication, and advanced tutoring and instructional strategies. With these revisions, the team expects to observe higher quality interaction between computer tutor and tutee and therefore increased effectiveness on learning gains beyond the interactivity plateau.

Research design and methods

The development of DeepTutor will follow an iterative systems engineering approach guided by a mixed-process software model. The team will use a mixed-model process and randomized controlled trials to evaluate the components of the system and the system as a whole. Students will be randomly assigned to participate in one of three conditions: DeepTutor, AutoTutor, or standard classroom instruction. All students will take a pretest, be exposed to one of three conditions, and then take a posttest. Information gleaned from the experimental findings will be used to revise and improve the design and implementation of the DeepTutor system.

Control condition

Students in the classroom condition will be taught the same content using a standard classroom method. Students in the AutoTutor condition will be taught the same content using the intelligent tutoring system AutoTutor.

Key measures

DeepTutor is intended to improve student outcomes in science relative to current state of the art in ITSs. Key measures include pretests and posttests related to the DeepTutor and AutoTutor interventions. Evaluations such as the Force Concept Inventory (FCI), a widely used multiple-choice test for evaluating understanding of introductory physics and the Force and Motion Conceptual Evaluation (FCMI), which is a similar multiple-choice test to the FCI, will be used. The main dependent variables will be the pretest scores, posttest scores, and time to complete the training problems. Researchers will also use both multiple-choice and essay tests to measure learning gains.

Data analytic strategy

An analysis of variance (ANOVA) will be conducted to detect any main effect of experimental condition or interactions. An analysis of covariance (ANCOVA) will be conducted to detect any differences or interactions among conditions with the pretest as a covariate.

People and institutions involved

IES program contact(s)

Christina Chhin

Education Research Analyst
NCER

Products and publications

Products: Products include a fully developed DeepTutor tutoring system. Published reports of the findings will also be produced.

Book chapter

Brawner, K., and Graesser, A. (2014). Natural Language, Discourse, and Conversational Dialogues Within Intelligent Tutoring Systems: A Review. In R. Sottilare, A.C. Graesser, X. Hu, and B. Goldberg (Eds.), Design Recommendations for Intelligent Tutoring Systems: Instructional Management, Volume 2 (pp. 189-204). Orlando, FL: Army Research Laboratory.

Cai, Z., Feng, S., Baer, W., and Graesser, A. (2014). Instructional Strategies in Trialogue-Based Intelligent Tutoring Systems. In R. Sottilare, A.C. Graesser, X. Hu, and B. Goldberg (Eds.), Design Recommendations for Intelligent Tutoring Systems: Instructional Management, Volume 2 (pp. 225-235). Orlando, FL: Army Research Laboratory.

Graesser, A.C., Dowell, N., and Clewley, D. (2017). Assessing Collaborative Problem Solving Through Conversational Agents. In A.A. von Davier, M. Zhu, and P.C. Kyllonen (Eds.), Innovative Assessment of Collaboration pp. 65-80, Springer International: Switzerland.

Graesser, A.C., Hu, X., Nye, B., and Sottilare, R. (2016). Intelligent Tutoring Systems, Serious Games, and the Generalized Intelligent Framework for Tutoring (GIFT). In H.F. O'Neil, E.L. Baker, and R.S. Perez (Eds.), Using Games and Simulation for Teaching and Assessment (pp. 58-79). New York: Routledge.

Graesser, A.C., Li, H., Feng, S. (2015). Constructing inferences in naturalistic reading contexts. In E. O'Brien, A. Cook, and R. Lorch, (Eds.), Inferences during Reading (pp. 290-320). Cambridge: Cambridge University Press.

Lintean, M., Rus, V., Cai, Z., Witherspoon-Johnson, A., Graesser, A.C., and Azevedo, R. (2012). Computational Aspects of the Intelligent Tutoring System MetaTutor. In P. McCarthy, and C. Boonthum-Denecke (Eds.), Applied Natural Language Processing: Identification, Investigation, and Resolution (pp. 247-260). Hershey, PA: IGI Global.

Mavrikis, M., D'Mello, S.K., Porayska-Pomsta, K., Cocea, M., and Graesser, A.C. (2010). Modeling Affect by Mining Students Interactions With Learning Environments. In C. Romero, S. Ventura, M. Pechenizkiy, and R.S. Baker (Eds.), Handbook of Educational Data Mining (pp. 231-244). New York: CRC Press.

McNamara, D.S., Jackson, G.T., and Graesser, A.C. (2010). Intelligent Tutoring and Games (ITaG). In Y.K. Baek (Ed.), Gaming for Classroom-Based Learning: Digital Role-Playing as a Motivator of Study (pp. 44-65). Hershey, PA: IGI Global.

Millis, K., Forsyth, C., Butler, H., Wallace, P., Graesser, A., and Halpern, D.F. (2011). Operation ARIES!: A Serious Game for Teaching Scientific Inquiry. In M.M.A. Oikonomou, and L. Jain (Eds.), Serious Games and Edutainment Applications (pp. 169-195). UK: Springer-Verlag.

Rus, V., and Niraula, N.B. (2012). Automated Detection of Local Coherence in Short Essays Based on Centering Theory. CICling 2012 (pp. 450-461). Berlin, Germany: Springer-Verlag.

Rus, V., Lintean, M., Graesser, A.C., and McNamara, D.S. (2012). Text-to-Text Similarity of Statements. In P. McCarthy, and C. Boonthum-Denecke (Eds.), Applied Natural Language Processing: Identification, Investigation, and Resolution (pp. 110-121). Hershey, PA: IGI Global.

Journal article, monograph, or newsletter

Graesser, A.C., and McNamara, D.S. (2011). Computational Analyses of Multilevel Discourse Comprehension. Topics in Cognitive Science, 3(2): 371-398.

Graesser, A.C., Forsyth, C., and Lehman, B. (2017). Two Heads May be Better Than One: Learning From Agents in Conversational Trialogues. Teacher College Record, 119(3). Full text

Graesser, A.C., Li, H., and Forsyth, C. (2014). Learning by Communicating in Natural Language With Conversational Agents. Current Directions in Psychological Science, 23(5): 374-380. Full text

Kopp, K., Britt, A., Millis, K., and Graesser, A. (2012). Improving the Efficiency of Dialogue in Tutoring. Learning and Instruction, 22(5): 320-330.

Nye, B.D., Graesser, A.C., and Hu, X. (2014). AutoTutor and Family: A Review of 17 Years of Natural Language Tutoring. International Journal of Artificial Intelligence in Education, 24(4): 427-469. Full text

Storey, J.K., Kopp, K.J., Wiemer, K., Chipman, P., and Graesser, A.C. (in press). Using AutoTutor to Teach Scientific Critical Thinking Skills. Behavior Research Methods.

Sullins, J., Craig, S.D., and Graesser, A.C. (2010). The Influence of Modality of Deep Reasoning Questions. International Journal of Learning Technology, 5(4): 378-387.

Proceeding

Lintean, M., and Rus, V. (2011). Dissimilarity Kernels for Paraphrase Identification. In Proceedings of the 24th International Florida Artificial Intelligence Research Society (FLAIRS) Conference (pp. 263-268). Menlo Park, CA: AAAI Press.

Lintean, M., and Rus, V. (2012). Measuring Semantic Similarity in Short Texts Through Greedy Pairing and Word Semantics. In Proceedings of the 25th International Florida Artificial Intelligence Research Society Conference (pp. 244-249). Marco Island, FL: Association for the Advancement of Artificial Intelligence.

Rus V., and Stefanescu D (2016). Toward Non-intrusive Assessment in Dialogue-Based Intelligent Tutoring Systems. In State-of-the-Art and Future Directions of Smart Learning. (pp. 231-241).

Rus V., Banjade R., Niraula N., Gire E., and Franceschetti D. (2017). A Study On Two Hint-level Policies in Conversational Intelligent Tutoring Systems. In Innovations in Smart Learning (pp. 171-181).

Rus V., Maharjan N., and Banjade R. (2017). Dialogue Act Classification In Human-to-Human Tutorial Dialogues. In Innovations in Smart Learning (pp. 183-186).

Rus, V., and Lintean, M. (2012). An Optimal Assessment of Natural Language Student Input Using Word-to-Word Similarity Metrics. In In Proceedings of the 11th International Conference on Intelligent Tutoring Systems (pp. 675-676). Chania, Crete, Greece: Springer-Verlag Berlin Heidelberg.

Related projects

iCODE: Investigating and Scaffolding Students' Code Comprehension Processes to Improve Learning, Engagement, and Retention

R305A220385

Questions about this project?

To answer additional questions about this project or provide feedback, please contact the program officer.

 

Questions about this project?

To answer additional questions about this project or provide feedback, please contact the program officer.

 

You may also like

Zoomed in IES logo
Workshop/Training

Data Science Methods for Digital Learning Platform...

August 18, 2025
Read More
Zoomed in IES logo
Workshop/Training

Data Science for Education (DS4EDU)

April 01, 2025
Read More
Zoomed in IES logo
Request for Applications

Education Research and Development Center Program ...

March 14, 2025
Read More