Description:

Purpose: This study is a response to the many calls from across the nation for implementing "an effective, realistic, affordable, and politically acceptable long-term approach to the well-known problems and opportunities of U.S. pre-K-16 STEM education" (National Science Board, May 2006). As an initial step to developing an intelligent computerized tutor for teaching scientific inquiry skills, the researchers will build an intelligent computerized tutor and information delivery system (Acquiring Research Investigative and Evaluative Skills, or ARIES). ARIES will teach key concepts of scientific inquiry by having the student hold conversations with two animated pedagogical agents, one a learner like the student, the other a tutor, as he or she solves a number of engaging problems in the social and physical sciences.

Project Activities: The success of ARIES on learning will be tested in university courses in which instructors will assign ARIES as homework in some class sections and not others. This study includes several randomized controlled experiments, each testing a particular aspect of ARIES. In the first experiment, the researchers examine the extent to which having students study problems from different disciplines produces transfer of concepts and skills from one situation to another. In the second experiment, the researchers compare the relative effectiveness of role-playing on different types of problems. In the third experiment, the researchers examine whether full dialog exchanges for the problems are necessary for deep learning. In the fourth experiment, the researchers test the effectiveness of reciprocal teaching using animated agents. In the fifth experiment, teachers will test whether the effectiveness of reciprocal teaching depends on the knowledge level of the animated teachable agent. All of these studies are designed to enable the research team to identify the features of the intelligent tutor system that are important for improving student learning. Although these initial studies will be conducted with college students, the researchers intend for the final version of the ARIES tutor to be a flexible low-cost solution for learning scientific inquiry in high school chemistry, biology or life science, and psychology courses.

Products: The products of this study include an interactive intelligent tutor system called ARIES that teaches inquiry skills to university students and published reports on the development and evaluation of ARIES.

Structured Abstract

Purpose: The researchers will develop and test an interactive intelligent tutor called ARIES that teaches inquiry skills to university students. ARIES will teach vital concepts of scientific inquiry by having the user hold conversations with two animated pedagogical agents as he or she solves a number of engaging problems in the social and physical sciences.

Setting: Research will be conducted in a rural university in Illinois.

Population: The majority of participants will be college students with a diverse race/ethnicity and SES backgrounds. More than 200 college students will participate over the duration of the study.

Intervention: Students will read an online text describing and explaining key concepts in scientific inquiry. To promote deep learning, students will teach an animated Other-Agent (a learner like the student) as the Guide-Agent (the tutor) looks on and makes suggestions. Later, the student will apply the learned concepts to problems that require the critical evaluation of realistically presented studies and causal claims. The Guide-Agent will tutor the student in natural language by using AutoTutor, a computerized tutor that mimics the dialog moves between human tutors and students. Students will be awarded points based on their answers and progress, giving the learning sessions the feel of a serious game. Every aspect of ARIES has been designed using well-known principles of learning. These include reciprocal teaching, self-explanation, spaced learning, practice at retrieval (testing effects), authentic learning, formative feedback, active responding, reflection, dialog interactivity, and variable encoding.

Research Design and Methods: This study includes five randomized controlled experiments, each testing a particular aspect of ARIES. The first experiment will examine the extent that having problems from different disciplines will produce transfer. The second experiment will compare the relative effectiveness of role-playing on different types of problems. The third experiment will examine whether full dialog exchanges for the problems are necessary for deep learning. The fourth experiment will test the effectiveness of reciprocal teaching using animated agents. The fifth experiment will test whether the effectiveness of reciprocal teaching depends on the knowledge level of the teachable agent.

Control Condition: A no-treatment control in university courses will be used.

Key Measures: The experiments will use many of the same measures to ascertain students' mastery of inquiry skills: multiple-choice questions, problems that require learners to evaluate scientific studies, and problems that require learners to design scientific studies. Pre-and post-versions of each will be prepared. These various tests assess different types of knowledge or retrieval processes. Multiple-choice questions tap knowledge via recognition memory; evaluate problems measure the degree that the student can recall and use the knowledge in an authentic way; design problems measure the active procedural knowledge relevant to scientific inquiry.

Data Analytic Strategy: The data will be analyzed by analysis of variance, analysis of covariance, and hierarchical linear regression.

Related IES Project: Center for the Study of Adult Literacy (CSAL): Developing Instructional Approaches Suited to the Cognitive and Motivational Needs for Struggling Adults (R305C120001)

Products and Publications

Book chapter

Brawner, K., and Graesser, A. (2014). Natural Language, Discourse, and Conversational Dialogues Within Intelligent Tutoring Systems: A Review. In R. Sottilare, A.C. Graesser, X. Hu, and B. Goldberg (Eds.), Design Recommendations for Intelligent Tutoring Systems: Instructional Management, Volume 2 (pp. 189–204). Orlando, FL: Army Research Laboratory.

Cai, Z., Feng, S., Baer, W., and Graesser, A. (2014). Instructional Strategies in Trialogue-Based Intelligent Tutoring Systems. In R. Sottilare, A.C. Graesser, X. Hu, and B. Goldberg (Eds.), Design Recommendations for Intelligent Tutoring Systems: Instructional Management, Volume 2 (pp. 225–235). Orlando, FL: Army Research Laboratory.

Graesser, A.C., and Forsyth, C.M. (2013). Discourse Comprehension. In D. Reisberg (Ed.), The Oxford Handbook of Cognitive Psychology (pp. 475–491). New York: Oxford University Press.

Graesser, A.C., and Lehman, B. (2012). Questions Drive Comprehension of Text and Multimedia. In M.T. McCrudden, J. Magliano, and G. Schraw (Eds.), Text Relevance and Learning From Text (pp. 53–74). Greenwich, CT: Information Age Publishing.

Graesser, A.C., and McNamara, D.S. (2012). Automated Analysis of Essays and Open-Ended Verbal Responses. In H. Cooper, P.M. Camic, D.L. Long, A.T. Panter, D. Rindskopf, and K.J. Sher (Eds.), APA Handbook of Research Methods in Psychology, Volume 1: Foundations, Planning, Measures, and Psychometrics (pp. 307–325). Washington, DC: American Psychological Association.

Graesser, A.C., Britt, A., Millis, K., Wallace, P., Halpern, D., Cai, Z., Kopp, K. and Forsyth, C. (2010). Critiquing Media Reports With Flawed Scientific Findings: Operation ARIES!, a Game With Animated Agents and Natural Language Trialogues. In J. Aleven, J. Kay, and J. Mostow (Eds.), Lecture Notes in Computer Science, 6095 (pp. 327–329). London: Springer.

Graesser, A.C., Chipman, P., and King, B.G. (2008). Computer-Mediated Technologies. In J.M. Spector, M.D. Merrill, J.J.G. VanMerriënboer, and M.P. Driscoll (Eds.), Handbook of Research on Educational Communications and Technology (3rd ed., pp. 211–224). London: Taylor and Francis.

Graesser, A.C., D'Mello, S., and Person, N.K. (2009). Meta-Knowledge in Tutoring. In D. Hacker, J. Donlosky, and A.C. Graesser (Eds.), Handbook of Metacognition in Education (pp. 361–382). Mahway, NJ: Taylor and Francis.

Graesser, A.C., Hu, X., Nye, B., and Sottilare, R. (2016). Intelligent Tutoring Systems, Serious Games, and the Generalized Intelligent Framework for Tutoring (GIFT). In H.F. O'Neil, E.L. Baker, and R.S. Perez (Eds.), Using Games and Simulation for Teaching and Assessment (pp. 58–79). New York: Routledge.

Graesser, A.C., Millis, K., D'Mello, S.K., and Hu, X. (2014). Conversational Agents can Help Humans Identify Flaws in the Science Reported in Digital Media. In D. Rapp, and J. Braasch (Eds.), Processing Inaccurate Information: Theoretical and Applied Perspectives From Applied Perspectives From Cognitive Science and the Educational Sciences (pp. 139–159). Cambridge, MA: MIT Press.

Graesser, A.C., Ozuru, Y., and Sullins, J. (2010). What Is a Good Question?. In M.G. McKeown, and L. Kucan (Eds.), Bringing Reading Research to Life (pp. 112–141). New York: Guilford Press.

Millis, K., Forsyth, C., Butler, H., Wallace, P., Graesser, A., and Halpern, D.F. (2011). Operation ARIES!: A Serious Game for Teaching Scientific Inquiry. In M.M.A. Oikonomou, and L. Jain (Eds.), Serious Games and Edutainment Applications (pp. 169–195). UK: Springer-Verlag.

Journal article, monograph, or newsletter

Forsyth, C.M., Graesser, A., Cai, Z., Butler, H., Halpern, D.F., Wallace, P., and Millis, K. (in press). Interrogating Aliens: Learning in a Game-Like Environment. Dialogue and Discourse.

Forsyth, C.M., Graesser, A.C., Pavlik, P., Cai, Z., Butler, H., Halpern, D.F., and Millis, K. (2013). Operation ARIES! Methods, Mystery, and Mixed Models: Discourse Features Predict Affect and Motivation in a Serious Game. Journal of Educational Data Mining, 5(1): 147–189.

Graesser, A.C. (2011). Learning, Thinking, and Emoting With Discourse Technologies. American Psychologist, 66(8): 746–757.

Graesser, A.C., and McNamara, D.S. (2010). Self-Regulated Learning in Learning Environments With Pedagogical Agents That Interact in Natural Language. Educational Psychologist, 45(4): 234–244.

Graesser, A.C., and McNamara, D.S. (2011). Computational Analyses of Multilevel Discourse Comprehension. Topics in Cognitive Science, 3(2): 371–398.

Graesser, A.C., Jeon, M., and Dufty, D. (2008). Agent Technologies Designed to Facilitate Interactive Knowledge Construction. Discourse Processes, 45(4): 298–322.

Graesser, A.C., Li, H., and Forsyth, C. (2014). Learning by Communicating in Natural Language With Conversational Agents. Current Directions in Psychological Science, 23(5): 374–380.

Halpern, D.F., Millis, K., Graesser, A.C. Butler, H., Forsyth, C., and Cai, Z. (2012). Operation ARA: A Computerized Learning Game That Teaches Critical Thinking and Scientific Reasoning. Thinking Skills and Creativity, 7(2): 93–100.

Kopp, K., Britt, A., Millis, K., and Graesser, A. (2012). Improving the Efficiency of Dialogue in Tutoring. Learning and Instruction, 22(5): 320–330.

Magliano, J.P., and Graesser, A.C. (2012). Computer-Based Assessment of Student-Constructed Responses. Behavioral Research Methods, 44(3): 608–621.

Storey, J.K., Kopp, K.J., Wiemer, K., Chipman, P., and Graesser, A.C. (in press). Using AutoTutor to Teach Scientific Critical Thinking Skills. Behavior Research Methods.

Proceeding

Butler, H.A., Forsyth, C., Halpern, D. F., Graesser, A.C., and Millis, K. (2010). Secret Agents, Alien Spies, and a Quest to Save the World: Operation ARIES! Engages Students in Scientific Reasoning and Critical Thinking. In Promoting Student Engagement. Volume 1: Programs, Techniques and Opportunities (pp. 286–292). Syracuse, NY: Society for the Teaching of Psychology.

Cai, Z., Graesser, A.C., Forsyth, C., Burkett, C., Millis, K., Wallace, P., Halpern, D., and Butler, H. (2011). Trialog in ARIES: User Input Assessment in an Intelligent Tutoring System. In Proceedings of the 3rd IEEE International Conference on Intelligent Computing and Intelligent Systems (pp. 429–433). Guangzhou: IEEE Press.

Forsyth, C., Butler, H.A., Graesser, A.C., Halpern, D.F., Millis, K., Cai, Z., and Wood, J. (2010). Higher Contributions Correlate With Higher Learning Aims. In Proceedings of the 3rd International Conference on Educational Data Mining (pp. 287–288). Pittsburgh: WordPress.

Forsyth, C.M., Pavlik, P., Graesser, A.C. Cai, Z., Germany, M., Millis, K., Butler, H., Halpern, D.F., and Dolan, R. (2012). Learning Gains for Core Concepts in a Serious Game on Scientific Reasoning. In Proceedings of the 5th International Conference on Educational Data Mining (pp. 172–175). Chania, Greece: International Educational Data Mining Society.