Purpose: The national focus on teacher quality has reinforced interest in the extent to which subject matter knowledge is a critical component of teacher quality. This interest has, in turn, led to efforts to identify measures of elementary teachers' knowledge about reading, and study the effects of this knowledge on students' learning. A literature review of measures of teachers' knowledge about reading, conducted by this study's researchers, indicated that no currently published studies have reported significant effects of elementary teachers' content knowledge on their students' gains in reading. One possible explanation for this finding is that the paper and pencil tests being used to measure teacher knowledge of reading do not represent well the complexity of reading instruction. The purpose of this study is to investigate whether a new measure which asks teachers to respond to videotaped reading lessons better captures the complexities of reading instruction and teachers' pedagogic content knowledge and is predictive of student achievement. A second purpose is to develop a video based professional development tool that could be used to improve teachers' knowledge about teaching reading. Videotaped reading lessons present dynamic and complex portraits of instruction; similarly, teachers' evaluations of videotaped reading lessons are an extremely promising medium for eliciting and measuring teacher knowledge.

Project Activities: This project will develop a measure that rates how effectively teachers' use their pedagogical content knowledge in teaching reading, and develop a closely related measure to assess the knowledge teachers use in analyzing reading lessons. The second measure will be embedded in a multimedia, interactive system of videotaped reading lessons (Case Studies of Reading Lessons, CSRL). It will incorporate a series of early elementary reading lessons with access to additional components to support understanding and evaluation of the lessons, including a transcript of the lesson, an interview with the teacher of the lesson, and the text/materials used in the lessons.

Structured Abstract

Setting: The study will take place in Michigan using teachers from urban districts there, primarily.

Population: First through third grade teachers from urban school districts in Michigan will be recruited, along with a volunteer sample of National Board Certified (NBC) applicants. The videotapes will be of other teachers working in classrooms located urban areas in Michigan with many children from poor and racially and ethnically diverse backgrounds. For some phases of the study, novice and expert teachers from these areas will be selected.

Intervention: The team will develop a measure of reading instruction knowledge using videoclips. This measure, Enacted Knowledge in Reading Lessons (EKRL), is intended to capture teacher pedagogical knowledge, and teachers' use of that knowledge during reading instruction. The team will also develop a multimedia, interactive system, Case Studies in Reading Lessons (CSRL), intended for use in professional development in reading instruction.

Research Design and Methods: The project involves three studies that use an iterative development and evaluation process. The first study is to develop and validate criteria for Enacted Knowledge in Reading Lessons (EKRL) and to develop a design concept for Case Studies of Reading Lessons (CSRL). Study 2 will have three aspects: (a) development of a fully functioning, interactive multimedia system for viewing and responding to CSRL; (b) comparison of experts and novices responses to questions following their interaction with case studies; and (c) evaluation of content, questions, formats of questions, and other feedback from cognitive interviews with teachers and consultant recommendations. Study 3 involves further development and evaluation of CSRL. The expert consultants will provide explanations of their evaluations of five reading lesson videoclips selected for CSRL on the basis of Study 2 results.

The first study will have three phases. In Phase 1, a measure of reading instruction knowledge using videoclips will be developed (EKRL). This will be done by reviewing the literature and consulting with nationally renowned reading experts to develop scoring criteria. Phase 2 will develop a design concept for the eventual multimedia, interactive system that provides the information needed to evaluate videoclips of reading lessons (CSRL). The plan is to include an interactive component through which the teacher can comment on and evaluate components of each lesson. Teachers can then have access to evaluations of the lessons prepared by literacy experts, offering them an opportunity to learn to look deeply into reading lessons and to engage in lesson analysis for purposes of self evaluation and professional growth. Consultants will review the design concept and provide recommendations for improvement, such as the kinds of information that would be useful for an observer in evaluating a videotaped reading lesson. Phase 3 will conduct a preliminary validation of EKRL, developed in Phase 1. Forty videotapes collected for previous IES-funded studies will be coded based on the coding scheme developed in phase 1. Trained video coders independently will apply the coding criteria to each lesson. When sufficient inter-rater reliability is achieved, the predictive validity of the overall measure will be tested by examining whether variation in lesson ratings of teachers in the videoclips predict significant variation in their students' year-end achievement, controlling for prior achievement, as well as other student, teacher, and school characteristics that were previously collected.

The second study involves collecting a second set of videotaped reading lessons from 20 Grade 1–3 teachers and developing a multimedia, interactive system for viewing and responding to reading lessons (CSRL). In this study 20 novice teachers and 20 National Board Certified middle childhood literacy teachers will rate the 20 lessons based on the criteria specified in the EKRL measure as an index of the teachers' pedagogical content knowledge, rank ordering the lessons, and determining the extent to which they distinguish experienced and novice teachers. The National Board Certified teachers will also participate in cognitive interviews.

In the third study, 50 first through third grade classroom teachers and 200 National Board Certified applicants for Early and Middle Childhood Generalist certification who teach similar grades will interact with CSRL, answering structured and open-ended questions about the lessons and providing ratings of lesson quality. Analysis of their performances will be used to examine issues of reliability and validity. Teachers will also complete surveys and interviews to gather information about the usability and practicality of the CSRL system.

Control Conditions. The measures being developed are informed and tested by the differences in responses of novice and expert teachers. Responses to CSRL will be compared for differences between teachers who receive National Board certification, teachers who applied but did not receive it, and teachers who have not applied to the National Boards.

Key Measures: The key outcome measure, Enacted Knowledge of Reading Lessons, (EKRL) is being developed for this study. Other measures being used during the development and validation process include: a previously developed Teacher Knowledge about Reading assessment; students' performances on the Iowa Test of Basic Skills, Reading Comprehension subtest for 2007 and 2008; and demographic information about the students, the teachers, and the schools (e.g., racial/ethnic characteristics).

Data Analytic Strategy: Some data will be analyzed quantitatively and other data qualitatively. Principal component factor analysis will help determine the internal structure of scale responses. Rasch item response theory (IRT) will be used to score the binary data and partial credit models (PCM) will be used to score the equal interval data. IRT analyses will provide item level difficulty and item to scale fit statistics. Hierarchical generalized linear models (HLGM) will be used to estimate rates of agreement between experienced and novice teachers.

Products and Publications

Journal article, monograph, or newsletter

Carlisle, J.F., Kelcey, B., Rosaen, C., Phelps, G., and Vereb, A. (2013). A Framework for Analysis of Case Studies of Reading Lessons. Journal of Education and Training Studies, 1(2): 224–238.

Rosaen, C.L., Carlisle, J.F., Mihocko, E., Melnick, A., and Johnson, J. (2013). Teachers Learning From Analysis of Other Teachers' Reading Lessons. Teaching and Teacher Education, 35: 170–184.

Vereb, A., Carlisle, J.F., and Mihocko–Bowling, E. (2015). Online Case Studies as a Professional Development Opportunity for Teachers of Elementary Reading. Journal of Technology and Teacher Education, 23(1): 107–131.

** This project was submitted to and funded under Teacher Quality: Reading and Writing in FY 2008.