Skip Navigation
Funding Opportunities | Search Funded Research Grants and Contracts

IES Grant

Title: Guru: A Computer Tutor that Models Expert Human Tutors
Center: NCER Year: 2008
Principal Investigator: Olney, Andrew Awardee: University of Memphis
Program: Education Technology      [Program Details]
Award Period: 3 years Award Amount: $1,858,176
Type: Development and Innovation Award Number: R305A080594
Description:

Purpose: There is substantial empirical evidence that one-on-one human tutoring is an effective supplement to typical classroom instruction. Yet, human tutors cannot be provided to every child—as there are simply not enough tutors. This project will develop Guru, a computer-based intelligent tutoring system, to assist high school students with biology learning.

Project: After developing prototypes of the individual software components of the intervention, researchers will conduct a series of design experiments focusing on implementation in real-world settings. Cycles of think-aloud protocols and eye tracking data from students will be collected to evaluate the usability and feasibility of the system. Expert human biology tutors will also aid in this evaluation and will provide feedback on the biology content. An evaluation will be conducted with the refined system to examine the promise of Guru for enhancing student learning.

Products: The intervention will consist of an expert computer tutor designed to support student achievement in biology.

Structured Abstract

Purpose: The purpose of the proposed project is to develop and evaluate Guru, an expert computer biology tutor, by modeling the strategies and dialog of expert human tutors. Once developed, Guru can be used to further understand the processes and mechanisms of expert tutoring by manipulating strategies and dialog moves within the tutor and observing associated student learning outcomes.

Setting: The setting is urban school districts in Memphis, Tennessee.

Population: The sample will include approximately 50 ninth-grade students in Memphis City Schools. In 2005 in this district, 75% of students were below proficient in science, 87% of students were African-American, 71% were eligible for free or reduced lunch, 79% were economically disadvantaged, and the graduation rate was 67%.

Intervention: Guru is an expert computer biology tutor, designed to be used as a supplement to classroom-based instruction in biology. When finalized, students will interact with the Guru animated agent by having a conversation with the tutor. During the course of the conversation, Guru and the student will work through biology topics and problems through a multimedia panel that presents movies, interactive diagrams, and other instructional media. As with expert human tutoring, an essential aspect of the Guru intervention is to precisely assess the student's answers and maintain an accurate model of the student's understanding. Guru is intended to promote educational attainment by targeting biology content that high school students must master in order to graduate in the state of Tennessee.

Research Design and Methods: The researchers will employ an Integrative Learning Design Framework to develop Guru. In this process, the components of the artificial intelligence system will be developed and then each component will be individually measured. Through the iterative design process, the expert human tutors and students will also participate in ongoing pilot tests and provide feedback on the biology content presented, as well as on the usability and feasibility of the system. The final pilot study will employ a counterbalanced within subjects repeated measures design with 30 students to test the promise of Guru for enhancing student learning outcomes.

Key Measures: Think-aloud protocols and eye tracking data from the target population will be collected and used to evaluate the usability and feasibility of the system, as well as to gather more information about users and needs. Multiple choice assessments will be used to measure learning of the biology content presented in the tutor. Measures used in the final pilot study will come directly from the Tennessee Gateway Biology Test.

Data Analytic Strategy: Data from interaction studies with expert human tutors, educators, and students will be used to verify that dialogs with the Guru expert tutor are consistent with that of a human expert tutor. This will be accomplished using correlational analyses of dialog as well as bystander Turing tests in which a human makes judgments on whether dialog was generated by a human or computer. The data collected during the final pilot study will be analyzed using an analysis of variance to test whether student scores are stable across assessments of topics in which no tutoring occurred, as well as the extent to which student's scores increase across assessments of topics for which they received tutoring.

Related IES Project: Center for the Study of Adult Literacy (CSAL): Developing Instructional Approaches Suited to the Cognitive and Motivational Needs for Struggling Adults (R305C120001)

Publications

Book chapter

Brawner, K., and Graesser, A. (2014). Natural Language, Discourse, and Conversational Dialogues Within Intelligent Tutoring Systems: A Review. In R. Sottilare, A.C. Graesser, X. Hu, and B. Goldberg (Eds.), Design Recommendations for Intelligent Tutoring Systems: Instructional Management, Volume 2 (pp. 189–204). Orlando, FL: Army Research Laboratory.

Cai, Z., Feng, S., Baer, W., and Graesser, A. (2014). Instructional Strategies in Trialogue-Based Intelligent Tutoring Systems. In R. Sottilare, A.C. Graesser, X. Hu, and B. Goldberg (Eds.), Design Recommendations for Intelligent Tutoring Systems: Instructional Management, Volume 2 (pp. 225–235). Orlando, FL: Army Research Laboratory.

Cai, Z., Graesser, A.C., Hu, X., & Cockroft, J. L. (2019). Self-improving components in conversational intelligent tutoring systems. In A. Sinatra, A.C. Graesser, X. Hu, K. Brawner and V. Rus (Eds.), Design Recommendations for Intelligent Tutoring Systems: Self-improving systems (Vol.7) (pp. 119–126). Orlando, FL: Army Research Laboratory.

Graesser, A.C., and D'Mello, S.K. (2012). Emotions During the Learning of Difficult Material. In B.H. Ross (Ed.), The Psychology of Learning and Motivation, Volume 57 (pp. 183–225). San Diego: Elsevier Academic Press.

Graesser, A.C., and Forsyth, C.M. (2013). Discourse Comprehension. In D. Reisberg (Ed.), The Oxford Handbook of Cognitive Psychology (pp. 475–491). New York: Oxford University Press.

Graesser, A.C., and Lehman, B. (2012). Questions Drive Comprehension of Text and Multimedia. In M.T. McCrudden, J. Magliano, and G. Schraw (Eds.), Text Relevance and Learning From Text (pp. 53–74). Greenwich, CT: Information Age Publishing.

Graesser, A.C., and McNamara, D.S. (2012). Automated Analysis of Essays and Open-Ended Verbal Responses. In H. Cooper, P.M. Camic, D.L. Long, A.T. Panter, D. Rindskopf, and K.J. Sher (Eds.), APA Handbook of Research Methods in Psychology, Volume 1: Foundations, Planning, Measures, and Psychometrics (pp. 307–325). Washington, DC: American Psychological Association.

Graesser, A.C., Baer, W., Feng, S., Walker, B., Clewley, D., Hays, D.P., Greenberg, D. (2015). Emotions in adaptive computer technologies for adults improving reading. In S. Tettegah and M. Gartmeier (Eds.), Emotions, Technology, Design, and Learning (pp. 3-25). New York: Elsevier.

Graesser, A.C., Keshtkar, F., and Li, H. (2014). The Role of Natural Language and Discourse Processing in Advanced Tutoring Systems. In T. Holtgraves (Ed.), The Oxford Handbook of Language and Social Psychology (pp. 491–509). New York: Oxford Handbooks Online.

Graesser, A.C., Feng, S., & Cai, Z. (2017). Two technologies to help adults with reading difficulties improve their comprehension. In E. Segers and P. Van den Broek (Eds.), Developmental perspectives in written language and literacy. In honor of Ludo Verhoeven (pp. 295-313). John Benjamin Publishing Company.

Graesser, A. C., Lippert, A. M., & Hampton, A. J. (2017). Successes and failures in building learning environments to promote deep learning: The value of conversational agents. In J. Buder & F. W. Hesse (Eds.), Informational Environments: Effects of Use, Effective Designs (pp. 273–298). Springer International Publishing. https://doi.org/10.1007/978-3-319-64274-1_12

Graesser, A.C., Millis, K., D'Mello, S.K., and Hu, X. (2014). Conversational Agents can Help Humans Identify Flaws in the Science Reported in Digital Media. In D. Rapp, and J. Braasch (Eds.), Processing Inaccurate Information: Theoretical and Applied Perspectives From Applied Perspectives From Cognitive Science and the Educational Sciences (pp. 139–159). Cambridge, MA: MIT Press.

Li, H., Shubeck, K. and Graesser, A. C. (2016). Using Technology in Language Assessment. In D., Tsagari, & J. V. Banerjee. (Eds.), Contemporary Second Language Assessment: Contemporary Applied Linguistics (Vol. 4, pp. 281-297). London, UK: Bloomsbury Academic.

Journal article, monograph, or newsletter

Baker, R.S., D'Mello, S.K., Rodrigo, M.T., and Graesser, A.C. (2010). Better to be Frustrated Than Bored: The Incidence, Persistence, and Impact of Learners' Cognitive-Affective States During Interactions With Three Different Computer-Based Learning Environments. International Journal of Human-Computer Studies, 68: 223–241.

D'Mello, S., and Graesser, A.C. (2010). Multimodal Semi-Automated Affect Detection From Conversational Cues, Gross Body Language, and Facial Features. User Modeling and User-Adapted Interaction, 20(2): 147–187.

D'Mello, S., Craig, S., and Graesser, A. (2009). Multi-Method Assessment of Affective Experience and Expression During Deep Learning. International Journal of Learning Technology, 4(3): 165–187.

D'Mello, S., King, B., Chipman, P., and Graesser, A.C. (2010). Towards Spoken Human-Computer Tutorial Dialogues. Human Computer Interaction, 23: 289–323.

D'Mello, S.K., and Graesser, A.C. (2009). Automatic Detection of Learner's Affect From Gross Body Language. Applied Artificial Intelligence, 23: 123–150.

D'Mello, S.K., Lehman, B., and Person, N. (2010). Monitoring Affect States During Effortful Problem Solving Activities. International Journal of Artificial Intelligence in Education, 20(4): 361–389. doi:10.3233/JAI-2010–012

D'Mello, S.K., Olney, A., and Person, N. (2010). Mining Collaborative Patterns in Tutorial Dialogues. Journal of Educational Data Mining, 2(1): 1–37.

D'Mello, S.K, and Graesser, A.C. (2012). Dynamics of Affective States During Complex Learning. Learning and Instruction, 22(2): 145–157.

Gholson, B., Witherspoon, A., Morgan, B., Brittingham, J.K., Coles, R., Graesser, A.C., Sullins, J., and Craig, S.D. (2009). Exploring the Deep-Level Reasoning Questions Effect During Vicarious Learning Among Eighth to Eleventh Graders in the Domains of Computer Literacy and Newtonian Physics. Instructional Science, 37(5): 487–493.

Graesser, A.C. (2009). Cognitive Scientists Prefer Theories and Testable Principles With Teeth. Educational Psychologist, 44(3): 193–197.

Graesser, A.C. (2011). Learning, Thinking, and Emoting With Discourse Technologies. American Psychologist, 66(8): 746–757.

Graesser, A.C. (2016). Conversations with AutoTutor help students learn. International Journal of Artificial Intelligence in Education, 26, 124 –132. Full text

Graesser, A.C., and McNamara, D.S. (2010). Self-Regulated Learning in Learning Environments With Pedagogical Agents That Interact in Natural Language. Educational Psychologist, 45(4): 234–244.

Graesser, A.C., and McNamara, D.S. (2011). Computational Analyses of Multilevel Discourse Comprehension. Topics in Cognitive Science, 3(2): 371–398.

Graesser, A. C., Cai, Z., Morgan, B., & Wang, L. (2017). Assessment with computer agents that engage in conversational dialogues and trialogues with learners. Computers in Human Behavior, 76, 607-616. Full text

Graesser, A.C., Li, H., and Forsyth, C. (2014). Learning by Communicating in Natural Language With Conversational Agents. Current Directions in Psychological Science, 23(5): 374–380. Full text

Graesser, A.C., Forsyth, C., and Lehman, B. (2017). Two Heads May be Better Than One: Learning From Agents in Conversational Trialogues. Teacher College Record, 119(3). Full text

Kopp, K., Britt, A., Millis, K., and Graesser, A. (2012). Improving the Efficiency of Dialogue in Tutoring. Learning and Instruction, 22(5): 320–330.

Lippert, A., Shubeck, K., Morgan, B., Hampton, A, & Graesser, A.C. (2020) Multiple Agent Designs in Conversational Intelligent Tutoring Systems. Technology, Knowledge and Learning, v25 n3 p443-463.

Louwerse, M.M., Graesser, A.C., McNamara, D.S., and Lu, S. (2009). Embodied Conversational Agents as Conversational Partners. Applied Cognitive Psychology, 23(9): 1244–1255.

Nye, B.D., Graesser, A.C., and Hu, X. (2014). AutoTutor and Family: A Review of 17 Years of Natural Language Tutoring. International Journal of Artificial Intelligence in Education, 24(4): 427–469. Full text

Olney, A. (2011). Large-Scale Latent Semantic Analysis. Behavior Research Methods, 43(2): 414–423.

Rus, V., McCarthy, P.M., McNamara, D.S., and Graesser, A.C. (2009). Identification of Sentence-to-Sentence Relations Using a Text Entailer. Research on Language and Computation, 7(2): 371–398.

Wiley, J., Goldman, S.R., Graesser, A.C., Sanchez, C.A., Ash, I.K., and Hemmerich, J.A. (2009). Source Evaluation, Comprehension, and Learning in Internet Science Inquiry Tasks. American Educational Research Journal, 46(4): 1060–1106.

Proceeding

Cai, Z., Li, H., Hu, X., & Graesser A. C. (2016). Can word probabilities from LDA be simply added up to represent documents? In T. Barnes, M. Chi, & M. Feng (Eds.), In Proceedings of the 9th International Conference on Educational Data Mining (pp. 577-578). Raleigh, North Carolina: EDM Society. Full text

Cai, Z., Siebert-Evernston, A., Eagan, B., Shaffer, D.W., Hu, X., & Graesser, A.C. (2019). nCoder+: A Semantic Tool for Improving Recall of nCoder Coding. In B. Eagan, M. Misfeldt, A. Siebert-Evenstone (Eds.), Advances in Quantitative Ethnography Proceedings of the First International Conference on Quantitative Ethnography (pp.41-54). Madison, WI: Springer, Cham.

Li, H., Cai, Z., & Graesser A. C. (2016). How good is popularity? Summary grading in crowdsourcing. In T. Barnes, M. Chi, & M. Feng (Eds.), Proceedings of the 9th International Conference on Educational Data Mining (pp. 430-435). Raleigh, North Carolina: EDM Society. Full text

Li, H., Cheng, C., Yu, Q., & Graesser A. C. (2015). The role of peer agent's learning competency in trialogue-based reading intelligent systems. In C. Conati, N. Heffernan, A. Mitrovic, M. F. Verdejo (Eds.), Proceedings of 17th International Conference on Artificial Intelligence in Education (pp. 694–697). Cham: Springer. Full text


Back