Project Activities
Building on two multi-agent adaptive learning technologies (MetaTutor and Betty's Brain) previously developed by the PI's, the research team will develop the SimSelf computer-based model. Following an iterative development process in Years 1 and 2, the team will collaborate with middle school science teachers to identify science content and problems, develop the SimSelf system in prototype form incorporating identified science content, and carry out informal field testing and revision. In Year 3, a pilot study will be conducted to ascertain the feasibility of classroom implementation and gather preliminary data as to the promise of SimSelf in improving student SRL skills and concept learning.
Structured Abstract
Setting
This study will take place at Vanderbilt University and McGill University, and in middle schools in Nashville, Tennessee and Montreal, Quebec, Canada.
Sample
Each phase of the initial intervention development during Year 1 will involve 20 seventh and eighth grade students. Six seventh and eighth grade science teachers will participate in each Module Evaluation Study in Year 1 and Year 2. Participants in the final pilot study during Year 3 will include about six middle school teachers and approximately 200 seventh and eighth graders from the Nashville, TN and Montreal, Quebec, Canada areas.
The researchers will develop three SimSelf modules which will consist of a suite of adaptive pedagogical agents and supporting services that monitor and scaffold student SRL skills and knowledge of science content necessary to effectively engage in meaningful learning tasks or to solve complex science problems. The proposed Peer and Tutor agents in SimSelf directly model students' SRL behaviors during system use and coordinate feedback and scaffolding among a set of tutor and peer agents. In general, by supporting student learning and problem solving using peer agents, students are hypothesized to be better able to gain both domain knowledge and SRL processes. Within the SimSelf model itself, students experience three phases of SRL development: (1) an observation phase in which strategies are introduced through vicarious learning, (2) an emulation phase in which students practice what they have learned in the observation phase through step-by-step guidance in problem solving, and (3) a self-controlled phase in which the student (with agent support) applies strategies learned previously to learning tasks and problems.
Research design and methods
This project team will use participatory design principles and cycles of iterative development to build, test, and evaluate three SimSelf modules and components in seventh and eighth grade classrooms. Early stages of the development in Years 1 and 2 will include four repeated cycles of teacher input to identify science content and problems, the development of the SimSelf system in prototype form incorporating the identified science content, usability testing, and module evaluation with middle school students; module refinement with middle school students; and module evaluation by science teachers. Research team members with expertise in various interdisciplinary methods and analytical tools will also contribute to the development cycles. These four cycles are repeated for each of the three modules and they will each take 6-12 months. Each cycle has different goals, number of participants, and different types of data collected and analyzed. In the last year of the grant, the research team will assess the effects of the system by conducting a pilot study using an experimental design in several seventh and eighth grade classrooms. As part of this pilot study, the team will also assess the feasibility of classroom implementation. The team will assess student concept learning using pre- and posttests to assess student SRL skills using modified adaptations of previously developed instruments.
Control condition
Students in the control condition are made aware of the same strategies, but no scaffolding is provided to support their learning.
Key measures
The researchers will use existing, researcher-developed measures of student learning and problem solving, as well as standardized self-report measures of motivation and affect such as the Motivation and Strategies Learning Questionnaire and the Achievement Emotions Questionnaire. The researchers will also use conventional measures drawn from Advanced Placement courses and achievement tests with content consistent with the American Association for the Advancement of Science Benchmarks for Scientific Literacy and the Tennessee state science standards. Process data from concurrent think-aloud protocols, eye-tracking, and affect detection and classification will be analyzed to study the deployment of SRL processes during learning and problem-solving and their correlation to learning and problem-solving outcomes. In addition, educational data mining techniques will be applied to log files of student activity to model and assess learning behaviors and SRL processes; determine efficacy and time spent planning, enacting, and reflecting during the various phases of learning; and understand how agent feedback and dialogue affect these processes. Facial detection and classification data will be used to study learning-centered emotions.
Data analytic strategy
Large amounts of quantitative and qualitative data will be analyzed using correlations, T-tests, analysis of variance, regression analyses, sequence mining techniques, and hidden Markov models. These models can provide an overview of common learning behaviors and strategies employed by an individual student or a set of students during learning, as well as the likelihood of transitioning between strategies (such as probing, checking, asking questions, reflecting on quiz results). Using hidden Markov models confirms the presence or absence of expected behaviors and identifies the interplay of these strategies as the students combine them in their learning activities; it also suggests further additions and refinements to the agent feedback for promoting SRL. Existing coding schemes will also be utilized and extended to organize, classify, and analyze the qualitative data collected from the iterative cycles of development and testing (e.g., quality of student-agent interactions during problem solving).
People and institutions involved
IES program contact(s)
Project contributors
Products and publications
Products will include SimSelf, a fully developed computer-based intervention comprised of three modules for seventh and eighth grade science students. Peer reviewed publications will also be produced.
Publications:
Journal article, monograph, or newsletter
Azevedo, R. (2014). Issues in Dealing With Sequential and Temporal Characteristics of Self- and Socially-Regulated Learning. Metacognition and Learning, 9(2): 217-228.
Azevedo, R. (2015). Defining and Measuring Engagement and Learning in Science: Conceptual, Theoretical, Methodological, and Analytical Issues. Educational Psychologist, 50(1): 84-94.
Biswas, G., Segedy, J.R., and Bunchongchit, K. (2016). From Design to Implementation to Practice a Learning by Teaching System: Betty's Brain. International Journal of Artificial Intelligence in Education, 26(1): 350-364.
Harley, J.M. and Azevedo, R. (2014). Toward a Feature-Driven Understanding of Students' Emotions During Interactions With Agent-Based Learning Environments: A Selective Review. International Journal of Gaming and Computer-Mediated Simulation, 6(3): 17-34.
Kinnebrew, J.S., Loretz, K.M., and Biswas, G. (2013). A Contextualized, Differential Sequence Mining Method to Derive Students' Learning Behavior Patterns. Journal of Educational Data Mining, 5(1): 190-219.
Kinnebrew, J.S., Segedy, J.R., and Biswas, G. (2014). Analyzing the Temporal Evolution of Students' Behaviors in Open-Ended Learning Environments. Metacognition and Learning, 9(2): 187-215.
Roscoe, R.D., Segedy, J.R., Sulcer, B., Jeong, H., and Biswas, G. (2013). Shallow Strategy Development in a Teachable Agent Environment Designed to Support Self-Regulated Learning. Computers & Education, 62: 286-297.
Segedy, J.R., Biswas, G., and Sulcer, B. (2014). A Model-Based Behavior Analysis Approach for Open-Ended Environments. Journal of Educational Technology and Society, 17(1): 272-282.
Segedy, J.R., Kinnebrew, J.S., and Biswas, G. (2013). The Effect of Contextualized Conversational Feedback in a Complex Open-Ended Learning Environment. Educational Technology Research and Development, 61(1): 71-89.
Segedy, J.R., Kinnebrew, J.S., and Biswas, G. (2015). Using Coherence Analysis to Characterize Self-Regulated Learning Behaviours in Open-Ended Learning Environments. Journal of Learning Analytics, 2(1): 13-48.
Related projects
Questions about this project?
To answer additional questions about this project or provide feedback, please contact the program officer.
