The Development of an Intelligent Pedagogical Agent for Physical Science Inquiry Driven by Educational Data Mining
Purpose: The National Research Councilís new framework for K-12 science education emphasizes the integration of scientific inquiry with disciplinary content knowledge. This integration is intended to help students learn to apply and transfer their science knowledge in more flexible ways. Students, however, continue to have difficulty with learning scientific inquiry. Recent studies suggest that tutoring students in scientific inquiry through the use of pedagogical agents is a promising, innovative approach for improving student learning in science. To that end, the current project will develop a pedagogical agent designed to assist students in learning inquiry in new science topics while also seamlessly integrating assessments with the instruction.
Project Activities: The current project will build on an existing suite of 12 inquiry microworlds for physical science by developing and refining a set of inquiry detectors to assess studentsí inquiry skills and behaviors and a pedagogical agent to present scaffolds for each inquiry skill. The researchers will assess the promise of the newly developed agent and revised system by conducting an experimental study comparing students who receive the pedagogical agent to those who do not, and measure the differences in inquiry skill acquisition within and across Physical Science topics.
Products: The products of this project include a fully developed suite of microworlds for Physical Science with an integrated pedagogical agent, and peer-reviewed publications.
Setting: The setting for this study includes diverse urban and suburban middle schools and an after school program in Massachusetts.
Sample: The sample will include middle school students drawn from an after school program and classrooms in three middle schools.
Intervention: Under previous grant awards, including a prior IES Goal 2 award, the researchers have successfully developed microworlds, inquiry tasks, and teacher report systems to assess middle school studentsí knowledge in physical, life and earth science. The current study will develop a pedagogical agent to be integrated into 12 previously developed physical science microworlds covering the topics of: state change; density; displacement; mass and weight; conservation of energy; parabolic motion; levers; elastic collisions, pendulum properties; and ramps. Within the microworld environment, students conduct inquiry by generating hypotheses, collecting data to test their hypotheses, interpreting the data, warranting claims, and communicating findings. The pedagogical agent will guide students through each step of the inquiry process and provide real-time scaffolding through validated assessments. The pedagogical agent scaffolds by delivering messages, using gestures, and illustrating through demonstrations. These scaffolds will be of four types: orienting, conceptual, procedural, and instrumental. The pedagogical agent will identify students who are struggling at specific inquiry skills and give them multi-level feedback targeted towards helping them understand both conceptual and procedural aspects of inquiry. Not only will the system provide individualized scaffolding to the student in real-time, it will provide teachers with rigorous, real-time assessment reports with which they can use to tailor instruction.
Research Design and Methods: An iterative development process will be used to develop, test, and revise the inquiry detectors, latent skill models, and scaffolds along with the pedagogical agent. Data from studentsí use of the microworlds will be collected and used to inform the development of the detectors for each inquiry skill. Inquiry detectors will be developed using a combination of knowledge-engineered and data-mined rules to model and assess studentsí inquiry skills and behaviors across tasks and microworlds/topics. The newly developed inquiry detectors will then be re-validated and tested with a new set of students. In addition, the researchers will also design and test the pedagogical agent. The scaffolds provided by the pedagogical agent will be pilot tested with small groups of 50 students from afterschool programs and middle schools. Student think-aloud data will be collected to determine which modifications are needed to improve the scaffolding. In the middle schools, the researchers will audiotape students using the microworlds and tutoring prompts so teachers can observe students and suggest scaffolding prompts for specific inquiry skills. By the end of this process, the researchers will have developed detectors capable of assessing each inquiry skill, pedagogical agents, and scaffolds within all physical science microworlds.
The researchers will assess the promise of the newly developed and revised system by conducting an experimental study comparing students who receive the pedagogical agent to those who do not, and measure the differences in inquiry skill acquisition within and across physical science topics. A new cohort of students drawn from three middle schools will participate in the study. Students will be randomly assigned within the same class to the control or experimental condition per physical science microworld. Every student will participate in both conditions, but across different microworlds. Measures of performance will be collected and used to determine if the scaffolding results in better performance on inquiry skills, both immediately and when scaffolding is no longer present.
Control Condition: For the pilot study, students in the control condition will receive the suite of microworlds without the scaffolding provided by the pedagogical agent.
Key Measures: The key measures in the study include assessments of studentsí inquiry skills for hypothesis generation, data interpretation and warranting claims, designing controlled experiments, and testing stated hypotheses.
Data Analytic Strategy: Logistic regressions will be used to analyze the impact of using the pedagogical agent on student performance within the inquiry microworlds, and the relative impact of receiving different amounts of scaffolding support on student performance. In addition, logistic regression analyses will be used to examine the degree to which the intervention leads to inquiry skills that can transfer across inquiry microworlds.
Related IES Projects: ASSISTment Meets Science Learning (AMSL) (R305A090170)
Project Website: http://slinq.org/
Gobert, J. (2014). Microworlds. In R. Gunstone (Ed.), Encyclopedia of Science Education (pp. 1). Netherlands: Springer.
Gobert, J.D. and Sao Pedro, M.A. (in press). Inq-ITS: Design Decisions Used for an Inquiry Intelligent System That Both Assesses and Scaffolds Students as They Learn. To appear in Rupp, A.A., and Leighton, J. (Co-Eds). Handbook of Cognition and Assessment . New York: Wiley/Blackwell.
Gobert, J.D., and Sao Pedro, M.A. (2017). Digital assessment environments for scientific inquiry practices. In A.A. Rupp, and J.P Leighton (Eds.), The Wiley Handbook of Cognition and Assessment: Frameworks, Methodologies, and Applications (pp. 508–534).
Journal article, monograph, or newsletter
Gobert, J., Sao Pedro, M., Baker, R.S., Toto, E., and Montalvo, O. (2012). Leveraging Educational Data Mining for Real Time Performance Assessment of Scientific Inquiry Skills Within Microworlds. Journal of Educational Data Mining, 15 (4): 153–185.
Gobert, J., Sao Pedro, M., Raziuddin, J., and Baker, R. (2013). From Log Files to Assessment Metrics: Measuring Students' Science Inquiry Skills Using Educational Data Mining. Journal of the Learning Sciences, 22 (4): 521–563.
Li, H., Graesser, A.C., and Gobert, J. (2017). ?????????????????[Where is embodiment hidden in the intelligent tutoring system?]. Journal of South China Normal University (Social Science Edition), 2, 79–91.
Gobert, J., Koedinger, K. and Raziuddin, J. (2013). Auto-Scoring Discovery and Confirmation Bias in Interpreting Data During Science Inquiry in a Microworld. In Proceedings of AIED (pp. 770–773). Berlin: Springer. doi:10.1007/978–3–642–39112–5_109
Li, H., Gobert, J., and Dickler, R (2017). Automated assessment for scientific explanations in on-line science inquiry. International Educational Data Mining Society (pp 214-219). Wuhan, China. Full text
Li, H., Gobert, J., and Dickler, R (2017). Dusting off the messy middle: Assessing students' inquiry skills through doing and writing.
Moussavi, R. and Gobert, J. (2016). Iterative Design, Development, and Evaluation of Scaffolds for Data Interpretation Practices during Inquiry. In Proceedings of the 12th International Conference of the Learning Sciences (pp. 1404). Singapore: International Society of the Learning Sciences.
Moussavi, R., Gobert, J., and Sao Pedro, M. (2016). The Effect of Scaffolding on the Immediate Transfer of Students' Data Interpretation Skills within Science Topics. In Proceedings of the 12th International Conference of the Learning Sciences (pp. 1002–1005). Singapore: International Society of the Learning Sciences.
Sao Pedro, M., Baker, R., and Gobert, J. (2013). Incorporating Scaffolding and Tutor Context into Bayesian Knowledge Tracing to Predict Inquiry Skill Acquisition. In Proceedings of the 6th International Conference on Educational Data Mining (EDM 2013) (pp. 185–193). Memphis, TN: International Educational Data Mining Society.
Sao Pedro, M., Baker, R., and Gobert, J. (2013). What Different Kinds of Stratification can Reveal About the Generalizability of Data-Mined Skill Assessment Models. In Proceedings of the 3rd Conference on Learning Analytics and Knowledge (pp. 190–195). New York: ACM. Full text