Funding Opportunities | Search Funded Research Grants and Contracts

IES Grant

Title: Automating the Measurement and Assessment of Classroom Discourse
Center: NCER Year: 2013
Principal Investigator: Nystrand, Martin Awardee: Board of Regents of the University of Wisconsin System
Program: Education Technology      [Program Details]
Award Period: 3 years (8/1/13- 1/31/16) Award Amount: $1,599,828
Goal: Measurement Award Number: R305A130030

Co-Principal Investigators: Arthur Graesser, Andrew Olney (University of Memphis); Sidney D’Mello (University of Notre Dame); Sean Kelly (University of Pittsburgh)

Purpose: The purpose of this project is to automate the Classroom Language Assessment System (CLASS 4.24), a computer program developed at the University of Wisconsin Center for Education Research. CLASS 4.24 is used to code classroom interactions between a teacher and his or her students, and (a) has successfully identified key instructional variables promoting reading achievement, (b) has been used to successfully support professional development, and (c) is currently used for teacher education at Michigan State and Ohio State Universities.  By automating CLASS 4.24, the team anticipates that the coding of classroom discourse will be radically simplified, thereby reducing its cost and increasing its potential use. CLASS 5.0 will facilitate scaling up the coding of classroom discourse, allowing classroom discourse to become a quantifiable variable for teacher education, professional development, and classroom research.

Project Activities:  The research team will adopt a systems-engineering approach involving iterative processes to design, test, revise, and retest the CLASS to produce CLASS 5.0, which will eventually function to automate the coding and results of CLASS 4.24. In order to automate the system three major technologies will be developed: (a) automatic speech recognition (ASR), (b) a question and speech act classifier (QSAC), and (c) macro-level discourse modeling. The system will be cross-validated against human coding of audio recordings.  The software system (QSAC and ASR) will be combined with a macro-level discourse model in order to identify higher-order latent categories of discourse structure in the sequence of speech acts.  Testing of usability and feasibility will occur in small focus groups and classroom environments.  Additional activities include the design and use of a sociological survey of the classroom context.

Products:  The product will be a software system that automatically transcribes and codes classroom discourse. This system can provide feedback to teachers that can inform their instruction, particularly in reading and literature classes. Peer-reviewed publications will also be produced.

Structured Abstract

Setting:  This project will take place in middle school classrooms, from a variety of settings (urban, suburban, and rural) in Wisconsin, and from a variety of classroom types (i.e., remedial, regular, college-prep, and untracked depending on school site).

Sample: Participants for this study include students in and teachers of grades 7–8.

Intervention:  The project goal is to develop an automated system that will measure classroom discourse.  CLASS 5.0 will measure dimensions of classroom discourse that influence the cognitively engaging dimensions of students’ literacy development as well as engagement in classroom instruction.  The system will record classroom discussions and analyze its content for discourse characteristics in order to provide feedback and guidance to teachers, so that they can in turn improve upon their dialogic instruction. A part of the system, the QSAC tool will identify authentic questions; classify student statements and their level of evaluation; and categorize the cognitive level of the statements.

Research Design and Methods: The research team will iteratively design, test, revise, and retest the automated system to produce CLASS 5.0, which will automate the coding and results of the current system, CLASS 4.24, which relies on human scoring. The work will proceed in parallel with two research teams—once at Wisconsin, the other at Memphis and Notre Dame. In Year 1, the Memphis/Notre Dame team will work with previously collected transcripts to create a version of CLASS 5.0 that works on transcripts without audio, while the Wisconsin team will record, transcribe, and code authentic sessions using high-quality microphones and phase microphone arrays. In Year 2, the Wisconsin team will collect more data and test the transcript-only system developed in Year 1. The team will test the usability and feasibility of the intervention with small focus groups and in classroom environments. The team will also design and carry out a sociological survey of classroom context. During this time, the Memphis/Notre Dame team will use the data Wisconsin collected in Year 1 to incorporate acoustic information into the existing models. At the end of Year 2, the Memphis/Notre Dame team will have produced the first fully functional version of CLASS 5.0. In Year 3, the Wisconsin team will conduct a pilot study of this version of CLASS 5.0 in classrooms. The goal of the pilot study is to investigate the relationship between students’ exposure to dialogic instruction and patterns of achievement growth. The Memphis/Notre Dame team will continue to refine CLASS 5.0, focusing both on performance and usability issues.

Control Condition: Due to the nature of this study design, there is no control condition.

Key Measures: Key measures of the system include comparison of the coding of discourse between the automated version of CLASS (CLASS 5.0) and the existing version (CLASS 4.24), which requires human coding.  Measures of student learning from Terra Nova include the reading test (grades 7 and 8) and the language arts and writing tests (grade 8); and a researcher-developed essay writing assessment, the coding of which examines the level of abstraction, coherence, and elaboration of argument.

Data Analytic Strategy: Researchers will use clustering, Hidden Markov Models, state transition networks, and other categorical time series analysis techniques to identify higher order latent categories of discourse structure in the sequence of speech acts. The research will use multilevel models of student achievement growth of literacy skills.  Student achievement data will be collected at two time points, allowing for change score models with multiple specifications to adjust for the relationship between initial status and achievement growth.