Skip Navigation
Funding Opportunities | Search Funded Research Grants and Contracts

IES Grant

Title: Acquiring Research Investigative and Evaluative Skills (ARIES) for Scientific Inquiry
Center: NCER Year: 2007
Principal Investigator: Millis, Keith Awardee: Northern Illinois University
Program: Cognition and Student Learning      [Program Details]
Award Period: 4 years Award Amount: $1,986,743
Goal: Development and Innovation Award Number: R305B070349

Purpose: This study is a response to the many calls from across the nation for implementing "an effective, realistic, affordable, and politically acceptable long-term approach to the well-known problems and opportunities of U.S. pre-K-16 STEM education" (National Science Board, May 2006). As an initial step to developing an intelligent computerized tutor for teaching scientific inquiry skills, the researchers will build an intelligent computerized tutor and information delivery system (Acquiring Research Investigative and Evaluative Skills, or ARIES). ARIES will teach key concepts of scientific inquiry by having the student hold conversations with two animated pedagogical agents, one a learner like the student, the other a tutor, as he or she solves a number of engaging problems in the social and physical sciences.

Project Activities: The success of ARIES on learning will be tested in university courses in which instructors will assign ARIES as homework in some class sections and not others. This study includes several randomized controlled experiments, each testing a particular aspect of ARIES. In the first experiment, the researchers examine the extent to which having students study problems from different disciplines produces transfer of concepts and skills from one situation to another. In the second experiment, the researchers compare the relative effectiveness of role-playing on different types of problems. In the third experiment, the researchers examine whether full dialog exchanges for the problems are necessary for deep learning. In the fourth experiment, the researchers test the effectiveness of reciprocal teaching using animated agents. In the fifth experiment, teachers will test whether the effectiveness of reciprocal teaching depends on the knowledge level of the animated teachable agent. All of these studies are designed to enable the research team to identify the features of the intelligent tutor system that are important for improving student learning. Although these initial studies will be conducted with college students, the researchers intend for the final version of the ARIES tutor to be a flexible low-cost solution for learning scientific inquiry in high school chemistry, biology or life science, and psychology courses.

Products: The products of this study include an interactive intelligent tutor system called ARIES that teaches inquiry skills to university students and published reports on the development and evaluation of ARIES.

Structured Abstract

Purpose: The researchers will develop and test an interactive intelligent tutor called ARIES that teaches inquiry skills to university students. ARIES will teach vital concepts of scientific inquiry by having the user hold conversations with two animated pedagogical agents as he or she solves a number of engaging problems in the social and physical sciences.

Setting: Research will be conducted in a rural university in Illinois.

Population: The majority of participants will be college students with a diverse race/ethnicity and SES backgrounds. More than 200 college students will participate over the duration of the study.

Intervention: Students will read an online text describing and explaining key concepts in scientific inquiry. To promote deep learning, students will teach an animated Other-Agent (a learner like the student) as the Guide-Agent (the tutor) looks on and makes suggestions. Later, the student will apply the learned concepts to problems that require the critical evaluation of realistically presented studies and causal claims. The Guide-Agent will tutor the student in natural language by using AutoTutor, a computerized tutor that mimics the dialog moves between human tutors and students. Students will be awarded points based on their answers and progress, giving the learning sessions the feel of a serious game. Every aspect of ARIES has been designed using well-known principles of learning. These include reciprocal teaching, self-explanation, spaced learning, practice at retrieval (testing effects), authentic learning, formative feedback, active responding, reflection, dialog interactivity, and variable encoding.

Research Design and Methods: This study includes five randomized controlled experiments, each testing a particular aspect of ARIES. The first experiment will examine the extent that having problems from different disciplines will produce transfer. The second experiment will compare the relative effectiveness of role-playing on different types of problems. The third experiment will examine whether full dialog exchanges for the problems are necessary for deep learning. The fourth experiment will test the effectiveness of reciprocal teaching using animated agents. The fifth experiment will test whether the effectiveness of reciprocal teaching depends on the knowledge level of the teachable agent.

Control Condition: A no-treatment control in university courses will be used.

Key Measures: The experiments will use many of the same measures to ascertain students' mastery of inquiry skills: multiple-choice questions, problems that require learners to evaluate scientific studies, and problems that require learners to design scientific studies. Pre-and post-versions of each will be prepared. These various tests assess different types of knowledge or retrieval processes. Multiple-choice questions tap knowledge via recognition memory; evaluate problems measure the degree that the student can recall and use the knowledge in an authentic way; design problems measure the active procedural knowledge relevant to scientific inquiry.

Data Analytic Strategy: The data will be analyzed by analysis of variance, analysis of covariance, and hierarchical linear regression.

Related IES Project: Center for the Study of Adult Literacy (CSAL): Developing Instructional Approaches Suited to the Cognitive and Motivational Needs for Struggling Adults (R305C120001)

Publications from this project:

Butler, H.A., Forsyth, C., Halpern, D.F., Graesser, A.C.,and Millis, K (2010). Secret Agents, Alien Spies, and A Quest To Save The World: Operation ARIES! Engages Students In Scientific Reasoning and Critical Thinking. In R. L. Miller, R. F. Rycek, E. Amsel, B. Kowalski, B. Beins, K. Keith, and B. Peden (Eds.)., Promoting Student Engagement. Volume 1: Programs, Techniques and Opportunities. Syracuse, NY: Society For The Teaching Of Psychology.

Cai, Z., Graesser, A.C., Forsyth, C., Burkett, C., Millis, K., Wallace, P., Halpern, D. and Butler, H. (2011). Trialog In ARIES: User Input Assessment In An Intelligent Tutoring System. In W. Chen and S. Li (Eds.), Proceedings Of The 3rd IEEE International Conference On Intelligent Computing and Intelligent Systems (pp.429–433). Guangzhou: IEEE Press.

Forsyth, C., Butler, H.A., Graesser, A.C., Halpern, D.F., Millis, K., Cai, Z., Wood, J. (2010). Higher Contributions Correlate With Higher Learning Aims. In R. S. J.D. Baker, A. Merceron, P. I. Pavlik (Eds.). Proceedings Of The3rd International Conference On Educational Data Mining (pp 287–288). Pittsburgh, PA: Wordpress.

Forsyth, C.M., Graesser, A., Cai, Z., Butler, H., Halpern, D.F., Wallace,P., and Millis, K. (In Press). Interrogating Aliens: Learning In A Game-Like Environment. Special Issue On Question Generation In Dialogue and Discourse.

Forsyth, C.M., Graesser, A.C., Pavlik, P., Cai, Z., Butler, H., Halpern, D.F., and Millis, K. (In Press). Operation ARIES! Methods, Mystery, and Mixed Models: Discourse Features Predict Affect and Motivation In A Serious Game. Journal Of Educational Data Mining.

Forsyth, C.M., Pavlik, P., Graesser, A.C. Cai, Z., Germany, M., Millis, K., Butler, H., Halpern, D.F., and Dolan, R. (2012). Learning Gains For Core Concepts In A Serious Game On Scientific Reasoning. In K.Yacef,O. ZaÔane, H. Hershkovitz, M. Yudelson, and J. Stamper (Eds.) Proceedings Of The 5th International Conference On Educational Data Mining (pp 172–175). Chania, Greece: International Educational Data Mining Society.

Graesser, A.C. (2011). Learning, Thinking, and Emoting With Discourse Technologies. American Psychologist, 66 (8): 746–757.

Graesser, A.C., and Forsyth, C.M. (2013). Discourse Comprehension. In D. Reisberg (Ed.), The Oxford Handbook Of Cognitive Psychology (pp. 475–491). New York, NY US: Oxford University Press.

Graesser, A.C, and Lehman, B. (2011). Questions Drive Comprehension Of Text and Multimedia. In M.T. Mccrudden, J.P. Magliano, G. Schraw (Eds.), Text Relevance and Learning From Text (pp. 53–74). Charlotte, NC US: IAP Information Age Publishing.

Graesser, A.C., and McNamara, D. (2010). Self-Regulated Learning In Learning Environments With Pedagogical Agents That Interact In Natural Language. Educational Psychologist, 45 (4): 234–244.

Graesser, A.C., and McNamara, D.S. (2011). Computational Analyses Of Multilevel Discourse Comprehension. Topics In Cognitive Science, 3 (2): 371–398.

Graesser, A.C., and McNamara, D.S. (2012). Automated Analysis Of Essays and Open-Ended Verbal Responses. In H. Cooper, P.M. Camic, D.L. Long, A.T. Panter, D. Rindskopf, K.J. Sher (Eds.), APA Handbook Of Research Methods In Psychology, Vol 1: Foundations, Planning, Measures, and Psychometrics (pp. 307–325). Washington, DC US: American Psychological Association.

Graesser, A.C., Britt, A., Millis, K., Wallace, P., Halpern, D., Cai, Z., Kopp, K. and Forsyth, C. (2010). Critiquing Media Reports With Flawed Scientific Findings: Operation ARIES!, A Game With Animated Agents and Natural Language Trialogues. In J. Aleven, J. Kay, and J. Mostow (Eds.). Lecture Notes In Computer Science, 6095 (pp.327–329). London: Springer.

Graesser, A.C., Chipman, P., and King, B.G. (2008). Computer-Mediated Technologies. In J.M. Spector, M.D. Merrill, J.J.G. Van MerriŽnboer, and M.P. Driscoll (Eds.), Handbook Of Research On Educational Communications and Technology (3rd Ed., pp. 211–224). London: Taylor and Francis.

Graesser, A.C., D'Mello, S., and Person, N. (2009). Meta-Knowledge In Tutoring. In D.J. Hacker, J. Dunlosky, A.C. Graesser (Eds.) , Handbook Of Metacognition In Education (pp. 361–382). New York, NY US: Routledge/Taylor and Francis Group.

Graesser, A.C., Jeon, M., and Dufty, D. (2008). Agent Technologies Designed To Facilitate Interactive Knowledge Construction. Discourse Processes, 45: 298–322.

Graesser, A.C., Ozuru, Y., and Sullins, J. (2010). What Is A Good Question? In M.G. Mckeown and L. Kucan (Eds.), Bringing Reading Research To Life (pp. 112–141). New York, NY: Guilford Press.

Halpern, D.F., Millis, K., Graesser, A.C., Butler, H., Forsyth, C., Cai, Z.(2012). Operation ARA: A Computerized Learning Game That Teaches Critical Thinking and Scientific Reasoning. Thinking Skills and Creativity, 7(2): 93–100.

Kopp, K.J., Britt, M., Millis, K., and Graesser, A.C. (2012). Improving The Efficiency Of Dialogue In Tutoring. Learning and Instruction, 22 (5): 320–330.

Magliano, J.P., and Graesser, A.C. (2012). Computer-Based Assessment Of Student-Constructed Responses. Behavior Research Methods, 44 (3): 608–621.

Millis, K., Forsyth, C., Butler, H., Wallace, P., Graesser, A. and Halpern, D.F. (2011). Operation ARIES!: A Serious Game For Teaching Scientific Inquiry. In M. Ma. A. Oikonomou and L. Jain. (Eds.), Serious Games and Edutainment Applications (pp. 169–195). UK: Springer-Verlag.

Storey, J.K., Kopp, K.J., Wiemer, K., Chipman, P., and Graesser, A.C. (In Press). Using Autotutor To Teach Scientific Critical Thinking Skills. Behavior Research Methods.