Co-Principal Investigator: Panayota Mantzicopoulos (Purdue University); Brian French (Washington State University)

Purpose: States across the nation are currently using various teaching effectiveness (TE) observational measures. Despite wide use of these measures for high-stakes decision making, significant yet unanswered questions about the accuracy of these assessments in documenting TE across content area, grade-level, or time, and with student outcomes beyond scores on standardized achievement tests remain. In this measurement project, researchers will study the psychometric properties of teaching effectiveness (TE) scores obtained in kindergarten classrooms across five widely used observation measures of teaching.

Project Activities: Researchers will investigate the psychometric properties of five currently widely used TE observational measures: 3 content-neutral (Classroom Assessment Scoring System; Framework For Teaching; the Marzano Comprehensive Observation Record), and 2 content-specific (Early Language & Literacy Classroom Observation, a K-3 tool for literacy; Mathematical Quality of Instruction for math).

Researchers will collect video data of lessons in three content areas recorded throughout the year. The video data will be collected for 15 weeks spaced evenly in Year 1 and 20 weeks in each of Years 2 and 3. Trained researchers will rate lessons using the TE measures. Different raters will use each TE measure for any given lesson. Student baseline data will be collected near the beginning of the year and outcome data at the end of the year from multiple respondents.

Products: The products of this project will be reliability and validity evidence for five widely used observation measures of teaching in the context of kindergarten classrooms. Peer-reviewed publications will also be produced.

Structured Abstract

Setting: Schools will be recruited from diverse locations in Indiana (including large city, mid-size city, urban fringe, small town, and rural).

Sample: In total, 100 kindergarten teachers, 145 classrooms, and 1160 kindergarteners will participate. In Year 1, 20 teachers will participate. In Year 2, 65 teachers will be recruited, at least 15 who participated in Year 1 and 50 new teachers. In Year 3, 60 teachers will be recruited, including 30 who participated in Year 2 and 30 new teachers. Approximately 1160 students – 8 students selected randomly selected from each of the 145 classrooms – will be recruited during the course of this project.

The sample average of participating schools will be equivalent to the state average in terms of student ethnicity (70% White, 12% Black or African American, 11% Hispanic or Latino, 5% Other, and 2% Asian) and free or reduced-cost lunch status (48%), and the school-level achievement. Within this sample, schools will differ widely in terms of these student characteristics and in location type. Classrooms will be heterogeneous in terms of teachers’ education and experience.

Assessment: Five widely used TE observational measures are the focus of this project. The Classroom Assessment Scoring System (CLASS) is intended to measure teacher-student interactions according to classroom social and emotional climate, organization, and quality of instructional support. The Framework For Teaching (FFT) is intended to measure teacher planning, instruction, classroom environment, and professional responsibilities. The Marzano Comprehensive Observation Record (COR) is intended to measure teacher behavior related to routine events, content, and on-the-spot strategies. The Early Language & Literacy Classroom Observation K-3 Tool (ELLCO K-3) is intended to measure key components of early reading and writing instruction, including classroom structure, curriculum, and classroom behaviors around language and discussion, books and book reading, and print and writing. The Mathematical Quality of Instruction (MQI) is intended to measure teacher provision of student learning opportunities and quality of mathematics instruction according to richness of mathematics, interactions with students, errors and imprecision, student participation and meaning-making, and connection of classroom work to mathematics during mathematics lessons.

Research Design and Methods: In order to investigate the psychometric properties of these five widely used TE observational measures, researchers will collect video data of lessons in three content areas recorded throughout the year. In Year 1, researchers will collect video data for 15 weeks spaced evenly across the year. In Years 2 and 3, the team will collect 20 weeks of observational data. Trained researchers will rate lessons using the TE measures. Different raters will use each TE measure for any given lesson. Student baseline data will be collected near the beginning of the year and outcome data at the end of the year from multiple respondents. For each measure, researchers will document: (1) the stability of TE scores across years, at different times within an academic year, and across multiple classroom observations; (2) the extent to which classroom observations predict changes in children’s achievement and motivation, for different numbers of observations and for observations at different times of the year; and (3) the extent to which child characteristics moderate the relation between TE and children’s achievement and motivation. For the 3 content-neutral assessments, researchers will also examine: (4) whether TE scores (including scores in subdomains of TE) are equally stable over a number of observations within and across years for each content area; and (5) the extent that relations between TE and changes in achievement and motivation vary by content area.

Control Condition: There is no control condition.

Key Measures: In addition to the teacher observational measures, the team will collect measures about student characteristics and academic outcomes. Demographic information (e.g., sex, ethnicity, free or reduced cost lunch status, special education status) will be collected from children’s school records. Academic achievement and behaviors will be measured using the Dynamic Indicators of Basic Early Literacy Skills (DIBELS) subtests of pre-reading and early reading; Woodcock-Johnson Tests of Achievement III (WJ-III) subtests for reading, writing, and math; a rating scale applied to student report card marks; and teacher ratings of student academic behaviors and social competence. Student motivation for learning will be measured using both student self-reported ratings and teacher-reported ratings. Student beliefs about the academic and social-emotional aspects of the classroom will be measured using student self-reported beliefs about the provision of opportunities for learning academic content, student report of teacher support for learning content in the content areas examined in the study, teacher report of the quality of the relationship with each student, student self-report of liking school, and student perceptions of teaching.

Data Analytic Strategy: Researchers will provide comprehensive psychometric evidence for each TE measure. Reliability evidence will include: (1) a generalizability study for each content area to examine variance components attributed to teachers, lessons, raters, the associated interactions, and measurement error; and (2) analyses to establish the number of lessons required to reach adequate reliability for each measure. Validity evidence will provide information on (3) internal structure via factor analysis, (4) relations to other variables (e.g., prediction), and (5) item-level measurement invariance, i.e., evidence that the TE measures employed in the study are measuring the same constructs and that scores have equivalent meaning across major subgroups (gender, ethnicity).