Description:

Previous Award Number: R305A110811
Previous Awardee: Boulder Language Technologies, Inc.

Co-Principal Investigators: Robert Talbot (University of Colorado, Denver), Michelene Chi (Arizona State University)

Purpose: Self-explanation has been shown repeatedly to be a key, contributing factor in deep learning of curriculum material. Research on tutoring benefits suggests that modeling good question asking and reasoning skills encourages deeper student comprehension, yet much of classroom instruction continues to use a teacher-led, didactic approach. This project will develop and pilot test a computer-based system and instructional method for simultaneously engaging all classroom students in self-explanation of science concepts. The system will enable teachers to pose deep questions frequently throughout a class and to immediately organize discussion around the diverse views present in student responses.

Project Activities: In the first two years of this project, master teachers will work with the research and software development teams to develop the self-explanation components, assessments, and teacher training materials. In Year 3, a pilot study will be conducted using a counterbalanced design in which student learning in Comprehension SEEDING classrooms is compared to student learning in control classrooms using either enhanced classroom discussion techniques or student response systems that utilize multiple-choice questions.

Products: This project will produce a fully developed computer-based system, instructional and teacher training materials, and assessments of deep science learning. A description of the intervention and evidence of its promise to improve science learning will be shared in peer-reviewed journals.

Structured Abstract

Setting: This study will take place in ten middle schools in the Boulder Valley School District, in Boulder, Colorado. Overall, approximately 25 percent of BVSD students are of a minority ethnicity and 17 percent qualify for free or reduced lunch.

Population: Ten sixth grade science teachers and approximately 1,000 students in their classes will participate in this study.

Intervention: The intervention will have three primary components: inquiry generation, self-explanation, and enhanced discussion. Inquiry generation occurs during the course of a classroom lecture or discussion. The teacher will ask students a deep reasoning question related to the content of the lecture or discussion that requires a constructed (free) response. Students then self-explain by submitting their constructed-response answers via wireless tablet computers which send their responses to the developed system. The system then analyzes the collection of responses using natural language processing in real-time as they are received and clusters the responses into conceptually similar groups. For a representative set of clusters, the system will select an answer prototypical of the group and display the cluster prototypes for the teacher and classroom to discuss. The system generates a set of deep questions, spanning a broad taxonomy of question types, for the teacher to consider. The teacher facilitates a discussion enhanced by varying conceptions and issues seen in the student responses, resulting in immediate feedback regarding key aspects of nearly every student's answer. This system moves beyond 'clickers,' a technology in current use that enables student classroom responses to multiple-choice questions posed during class presentations by facilitating constructed responses to complex questions.

Research Design and Methods: The system components will be developed and tested in an iterative, user-centered design and development process with embedded feasibility testing. Focus groups and interviews will be conducted with representative teachers and students. The purpose of the focus groups will be to present plans, mock-ups, and prototypes of user tasks to elicit design recommendations. The pilot study in Year 3 will utilize a counterbalanced design comprised of two control and two treatment conditions to provide evidence of the important components of Comprehension SEEDING. In control condition 1, teachers will use Questioning the Author and standard formative assessment techniques. Questioning the Author is a dialog-based approach that engages students in conceptual questions. In control condition 2, teachers will use classroom response technology ('clickers') and technology enhanced formative assessment tools designed to guide instruction through analysis of student responses to multiple-choice questions. In treatment condition 1, teachers will use system-supported self-explanation in which teacher-generated questions are posed to the classrooms and students provide answers using the Comprehension SEEDING system. In treatment condition 2, teachers will use both system-supported self-explanation and system-supported enhanced discussion that clusters student responses and provides guidance for further discussion.

Control Condition: The pilot study will compare student learning in classrooms using the intervention to two control conditions: classrooms using Questioning the Author, and classrooms using classroom response technology (also known as 'clickers') to elicit answers to multiple choice questions from students.

Key Measures: A new scenario-based measure of deep learning in physical science will be developed to measure student outcomes. Researchers will develop a measure of teacher skill in facilitating classroom discussion based on coding of transcripts of teacher and student talk during class. The Reformed Teaching Observation Protocol (RTOP), will be used to characterize the degree to which teachers are student-centered in their instruction. The RTOP consists of five subscales: Lesson Design and Implementation, Propositional Pedagogic Knowledge, Procedural Pedagogic Knowledge, Communicative Interactions, and Student-Teacher Relationships.

Data Analytic Strategy: Qualitative analyses of information collected in interviews, usability studies, teacher and student talk, and classroom observations will be used to refine the intervention and to verify its feasibility. Coding of qualitative data collected during the development activities will focus on identifying both a priori and emergent codes that focus on types of questions asked, frequencies of question types, and how the teacher responds to students' questions and comments. Data in the pilot study will be analyzed using repeated measures analysis of variance, as well as structural equation modeling to assess the promise of the intervention.

Products and Publications

Book chapter

Tomar, A., and Nielsen, R.D. (2013). Affective—Behavioral—Cognitive Learner Modeling. In R. Sottilare, A. Graesser, X. Hu, and H. Holden (Eds.), Design Recommendations for Intelligent Tutoring Systems: Learner Modeling, Volume I (pp. 75–86). Orlando, FL: U.S. Army Research Laboratory.

Journal article, monograph, or newsletter

Chi, M.T.H., and VanLehn, K.A. (2012). Seeing Deep Structure From the Interactions of Surface Features. Educational Psychologist, 47(3): 177–188.

Proceeding

Dzikovska, M.O., Nielsen, R.D., Brew, C., Leacock, C., Giampiccolo, D., Bentivogli, L., Clark, P, Dagan, I., and Dang, H.T. (2013). SemEval-2013 Task 7: The Joint Student Response Analysis and 8th Recognizing Textual Entailment Challenge. In Proceedings of the Second Joint Conference on Lexical and Computational Semantaics (*SEM 2013), 7th International Workshop on Semantic Evaluation (SemEval 2013). Atlanta, GA: Association for Computational Linguistics.

Godea, A., Bulgarov, F., and Nielsen, R.D. (2016). Automatic Generation and Classification of Minimal Propositions in Educational Systems. In 26th International Conference on Computational Linguistics (pp. 3226–3236). Osaka, JA: ACL Anthology.

Mazidi, K., and Nielsen, R.D. (2014). Linguistic Considerations in Automatic Question Generation. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, Volume 2 (pp. 321–326). Baltimore: Association for Computational Linguistics.

Mazidi, K., and Nielsen, R.D. (2014). Pedagogical Evaluation of Automatically Generated Questions. In Proceedings of the 12th International Conference on Intelligent Tutoring Systems (pp. 294–299). Honolulu: Springer.

Mazidi, K., and Nielsen, R.D. (2015). Leveraging Multiple Views of Text for Automatic Question Generation. In Proceedings of the 17th International Conference on Artificial Intelligence in Education (pp. 257–266). Madrid, Spain: Springer International Publishing.

Myroslava, D., Nielsen, R.D. and Brew, C. (2012). Towards Effective Tutorial Feedback for Explanation Questions: A Dataset and Baselines. In Proceedings of the 2012 Conference of the North American Association for Computational Linguistics: Human Language Technologies (2012 NAACL:HLT) (pp. 200–210). Montreal, Canada: Association for Computational Linguistics.

Paiva, F., and Nielsen, R.D. (2014). Clustering Constructed Responses for Formative Assessment in Comprehension SEEDING. In Proceedings of the 12th International Conference on Intelligent Tutoring Systems, Young Researchers Track (pp. 686–688). Honolulu: Springer.

Paiva, F., Glenn, J., Mazidi, K., Talbot, R., Wylie, R., Chi, M., Dutilly, E., Helding, B., Lin, M., Trickett, S., and Nielsen, R.D. (2014). Comprehension SEEDING: Comprehension Through Self Explanation, Enhanced Discussion, and Inquiry Generation. In Proceedings of the 12th International Conference on Intelligent Tutoring Systems (pp. 283–293). Honolulu: Springer.

Wylie, R., Chi, M.T.H., Talbot, R., Dutilly, E., Trickett, S., Helding, B., and Nielsen, R.D. (2014). Comprehension SEEDING: Providing Real—Time Formative Assessment to Enhance Classroom Discussion. In Proceedings of the 11th International Conference of the Learning Sciences, Volume 1 (pp. 1527–1528). Boulder, CO: International Society of the Learning Sciences.

Wylie, R., Helding, B., Talbot, R., Chi, M.T.H., Trickett, S., and Nielsen, R.D. (2014). Using Log Data to Predict Response Behaviors in Classroom Discussions. In Proceedings of the 12th International Conference on Intelligent Tutoring Systems (pp. 670–671). Honolulu: Springer.