Description:

Co-Principal Investigator: Lyon, Christine; Joe, Jilliam

Purpose: The use of an observation protocol by teachers is a potential leverage point for the improvement of teaching practice. However, there is currently limited research on the use of observation protocols by teachers, especially those designed to support improvements in formative assessment practice through peer observation and feedback. Researchers will refine the initial version of an observation system — the Formative Assessment Rubrics, Reflection, and Observation Protocol (FARROP) system (i.e., rubrics and online training materials) owned by Council of Chief State School Officers (CCSSO) but licensed under creative commons and available publicly. Teachers will use this system in a low-stakes context to observe and provide feedback to their peers on their formative assessment practices. Researchers will evaluate validity and reliability evidence for the use of the scores and inferences produced by the observation protocol.

Project Activities: Researchers will revise and evaluate an observation system designed to support peer observation and feedback. In Years 1 and 2, researchers will examine the feasibility of the system and perceptions of peer feedback. In Years 2 and 3, researchers will examine scoring, generalizability, extrapolation, and implications of scores.

Products: Researchers will produce a fully developed and validated measure of teacher formative assessment skills and peer-reviewed publications. They will communicate results to practitioners through the Formative Assessment for Students and Teachers (FAST) State Collaborative on Assessment and Student Standards (SCASS) with Council of Chief State School Officers (CCSSO); practitioner publications such as Education Week and The Learning Professional; and the Educational Testing Service (ETS) website.

Structured Abstract

Setting: In Years 1 and 2, school districts in New Jersey and Pennsylvania will participate in the project. In Year 3, school districts from Idaho, Michigan, and Oregon will participate.

Sample: Sixth through eighth grade English Language Arts teachers and their students will participate in this project. Six teachers in Year 1 and 16 teachers in Year 2 will participate; these teachers will be from up to 4 school districts across 2 states. In Year 3, 99 teachers and their approximately 1,980 students from 8 school districts across 3 states will participate.

Assessment: Researchers will design the observation system to support peer observation and feedback. The system includes an observation protocol focused on formative assessment (i.e., FARROP), online training modules to ensure accurate and reliable scoring, and online scoring system.

Research Design and Methods: In Years 1 and 2, researchers will collect data (e.g., video) about feasibility of the system and perceptions of peer feedback. In Years 2 and 3, videos will be scored by peer observers and external FARROP raters to examine scoring, generalizability, extrapolation, and implications of scores. In the Year 2 pilot test, participating teachers will each record 3 videos and receive peer feedback. A generalizability and dependability study will inform the Year 3 field test design. In the Year 3 field test, participating teachers will record 3 pre-intervention, 6 intervention, and 3 post-intervention videos (12 videos).

Control Condition: Due to the nature of the project, there is no control condition.

Key Measures: The research team will measure student achievement using the ELA Smarter Balanced assessment data. They will assess teacher practice using the FARROP and 2 subdomains of the Danielson's Framework for Teaching (i.e., the Instructional Domain and Classroom Environment Domain).

Data Analytic Strategy: Researchers will use agreement statistics to examine the similarity of scores between raters. In addition, they will use qualitative analysis to examine the usability of peer feedback and a linear mixed model to examine the relationship between student achievement and observation scores.