Project Website: https://wp0.vanderbilt.edu/oele/bettys-brain/

Purpose: The purpose of this project was to develop and test the use of teachable agents to improve students' metacognition, and to examine whether improving middle school students' metacognitive skills leads to improved science learning. Teachable agents are software programs in which students teach a computer agent to understand a concept and may be well suited to improving students' metacognitive skills. Metacognition involves two component processes: (a) the ability to monitor one's cognitive activities (e.g., detect comprehension failure), and (b) the ability to take appropriate regulatory steps when a problem has been detected (e.g., slowing down while reading difficult material, asking a teacher for help). These processes are critical for supporting students' abilities to solve problems and to learn.

Project Activities: First, the research team developed two curriculum units about ecosystems and created a student interface and a teachable agent named Betty. Next, the research team conducted studies to test the extent to which teachable agents help students learn to monitor their own and other people's problem solving and learning.

Products The products from this study included a teachable agent system called Betty's Brain, and published papers. In Betty's Brain, a teaching agent named Betty is taught using a concept map representation. Students use a point-and-click interface to teach her about entities, such as fish and algae, and their relations (e.g., fish consume dissolved oxygen, algae replenish it).

Key Outcomes: The main features of the intervention and findings of the project's pilot study are as follows:

• Use of self-regulated learning (SRL) supportive tools is positively correlated with learning outcomes (Roscoe, Segedy, Sulcer, Jeong, and Biswas, 2013).
• The use of promising strategy patterns weakened over time and students focused on a limited set of strategies at the cost of other beneficial strategies (Roscoe, Segedy, Sulcer, Jeong, and Biswas, 2013).

Structured Abstract

Setting: Schools were located in Tennessee and in California.

Sample: There were approximately 150 students in the 5th grade that participated in this research.

Intervention: Betty's Brain is a system that uses a teachable agent to support students' metacognition skills within the context of science. It includes two month-long curriculum units about ecosystems, one on climate change and one on thermoregulation. Students teach Betty, a teaching agent, using a concept map representation. Students use a point-and-click interface to teach her about concepts such as fish and algae and their relations. Once taught, Betty uses qualitative reasoning methods to answer questions. Learning by teaching is implemented as three primary components: (a) teach Betty using a concept map, (b) query Betty with your own questions to see how much she has understood, and (c) quiz Betty with a provided test to see how well she does on questions the student may not have considered. Students reflect on Betty's answers and revise their own knowledge as they make changes to the concept maps to teach Betty better. The system also includes a mentor who grades Betty's quiz answers and keeps track of how well she is performing. Additionally, the mentor monitors students' actions and guides them as needed.

Research Design and Methods: First, the research team developed the two curriculum units and the interface for Betty's Brain. Once those were developed, the research team investigated the potential for Betty's Brain to improve student metacognition in the form of self-monitoring and taking actions when warranted. Students were randomly assigned to one of three conditions: self-regulated learning, intelligent tutoring, or intelligent tutoring-self. In the self-regulated learning condition, students completed lessons by teaching their agent, Betty, and monitoring her performance. In the intelligent tutoring condition, students were told by the mentor agent to construct a concept map, and the mentor provided feedback. In this condition, students did not teach an agent, they were learning for themselves. The intelligent tutoring-self condition tested the hypothesis that outward monitoring provides a better support for learning to self-monitor than just instructions to self-monitor. Thus, students in the intelligent tutoring-self condition learned in the same manner as in the intelligent tutoring condition, except the mentor provided metacognition prompts. In addition to completing the two units, students in all conditions completed pre-tests and post-tests.

Control Condition: The researchers gathered benchmark data from 5th-grade classes at other schools that covered the same topics but did not use Betty's Brain.

Key Measures: Conventional measures were drawn from textbooks and achievement tests (e.g., TCAP), with content consistent with the AAAS Benchmarks for Scientific Literacy. The research team also analyzed the data from computer log files, including data such as how often students consulted quiz questions and content resources, how often they asked for help from the mentor, and how often they tested and revised their concept map.

Data Analytic Strategy: The research team used multilevel modeling to test the promise of Betty's Brain for improving students' metacognition and science achievement and correlational analysis to identify the relationship between students' behaviors within Betty's Brain and learning outcomes.

Related IES Projects: SimSelf: A Simulation Environment Designed to Model and Scaffold Learners' Self-Regulation Skills to Optimize Complex Science Learning (R305A120186)

Products and Publications

Book chapter

Hogyeong, J., Gupta, A., Roscoe, R., Wagster, J. Biswas, G., and Schwartz, D. (2008). Using Hidden Markov Models to Characterize Student Behaviors in Learning-By-Teaching Environments. Lecture Notes in Computer Science: Intelligent Tutoring Systems (pp. 614–625). Berlin: Springer.

Schwartz, D.L., Blair, K.P., Biswas, G., Leelawong, K., and Davis, J. (2007). Animations of Thought: Interactivity in the Teachable Agents Paradigm. In R. Lowe, and W. Schnotz (Eds.), Learning With Animation: Research and Implications for Design (pp. 114–140). Cambridge: Cambridge University Press.

Schwartz, D.L., Chase, C., Wagster, J., Okita, S., Roscoe, R., Chin, D., and Biswas, G. (2009). Interactive Metacognition: Monitoring and Regulating a Teachable Agent. In D.J. Hacker, J. Dunlosky, and A.C. Graesser (Eds.), Handbook of Metacognition in Education (pp. 340–358). New York: Routledge.

Journal article, monograph, or newsletter

Blair, K., Schwartz, D.L., Biswas, G., and Leelawong, K. (2007). Pedagogical Agents for Learning by Teaching: Teachable Agents. Educational Technology, 47(1): 56–61.

Chase, C., Chin, D.B., Oppezzo, M., and Schwartz, D.L. (2011). Teachable Agents and the Protege Effect: Increasing the Effort Towards Learning. International Journal of Science Education and Technology, 18(4): 334–352.

Chin, D.B., Dohmen, I.M., Cheng, B.H., Oppezzo, M.A., Chase, C.C., and Schwartz, D.L. (2010). Preparing Students for Future Learning With Teachable Agents. Educational Technology Research and Development, 58(6): 649–669.

Leelawong, K., and Biswas, G. (2008). Designing Learning by Teaching Agents: The Betty's Brain System. International Journal of Artificial Intelligence in Education, 18(3): 181–208.

Lindgren, R., and Schwartz, D.L. (2009). Spatial Learning and Computer Simulations in Science. International Journal of Science Education, 31(3): 419–438.

Roscoe, R.D., Segedy, J.R., Sulcer, B., Jeong, H., and Biswas, G. (2013). Shallow Strategy Development in a Teachable Agent Environment Designed to Support Self-Regulated Learning. Computers & Education, 62: 286–297.

Proceedings

Biswas, G., Schwartz, D., and Catley, K.M. (2008). Enhancing Learning Using Adaptive Computerized Tutoring in K–12 Settings. In B.C. Love, K. McRae, and V.M. Sloutsky (Eds.), Proceedings of the 30th Annual Conference of the Cognitive Science Society (pp. 695–696). Washington, DC: Cognitive Science Society.

Jeong, H., and Biswas, G. (2008). Mining Student Behavior Models in Learning-By-Teaching Environments. In Proceedings of the 1st International Conference on Educational Data Mining (pp. 127–136). Montreal, Canada.