Description:

Co-Principal Investigators: Amy Cassata-Widera and Jeanne Century

Purpose: Careful description and measurement of fidelity of implementation is essential in order to know which components of reform-based mathematics and science programs bolster or hinder student performance, or to determine the differential effects of incomplete or incorrect implementation of instructional materials. The determination of whether instructional materials have been adequately and faithfully implemented necessitates reliable and valid indicators of the extent, quality, and type of the implementation of the materials. However, there are few available tools to measure these factors. The goal of this project is to expand and validate a set of existing instruments for measuring the fidelity of implementation (FOI) of reform-based science and mathematics instructional materials. A suite of FOI instruments were developed, pilot tested and field tested through a grant from the National Science Foundation. Building on this foundational work, the present study proposes: (1) to revise and establish validity and reliability within and across three classroom-level FOI instruments (Teacher Instructional Questionnaire, Teacher Instructional Logs, Teacher Instructional Observation); and (2) to develop a student questionnaire to complement and triangulate constructs measuring student engagement within the existing instruments.

Project Activities: In the first year of the project, the team will develop the items for the Student Questionnaire (SQ), beginning with a literature and expert review process. Once the items are developed they will be piloted and then field tested with 300 students. Based upon the results of the pilot and field test, these items will be revised. At the same time, the team will revise and refine the teacher-level instruments, and will train the classroom observers. In the second year, the teacher and student instruments will be administered and student achievement data will be collected. In the second and third year, the team will complete a series of data analyses intended to assess the reliability and validity of the teacher and student FOI measures.

Products: Products from this project will include fully developed and validated classroom-level FOI instruments (Teacher Instructional Questionnaire, Teacher Instructional Logs, and Teacher Instructional Observation) and an SQ. In addition, final products will include a technical manual and revised user's guide. Peer-reviewed publications will also be produced.

Structured Abstract

Setting: This study will take place in 50 urban and suburban schools in 5 school districts including diverse geographic locales.

Population: Approximately 900 K–5th grade teachers will participate in the validation of the Teacher Instructional Questionnaire. In addition, a random subsample of 300 teachers will be selected to help validate the Teacher Instructional Observation Protocol and the Teacher Instructional Log. Approximately 4,500 students (grades 3–5) will complete two SQ's for math and science (10 students per classroom, 450 classrooms).

Intervention: A suite of fidelity of implementation (FOI) instruments will be developed, revised, and validated with the goal of enabling rigorous, specific, and systematic analysis of reform-based mathematics and science interventions. These instruments are grounded in a conceptual framework focusing on the critical components of a program or intervention. Critical components are defined as the key elements of a program that are explicitly or implicitly identified by the program's developer. With the critical components approach in mind, fidelity of implementation is further delineated into structural and process domains. Specifically, the FOI instruments to be developed and validated will organize program elements into two broad categories: (1) structural critical components, and (2) instructional critical components. Structural critical components include the procedural elements of a program that focuses on what the teacher needs to do, along with the developers' expectation for what the teacher need to know to implement the program with fidelity. Instructional critical components include the developers' expectations about the behaviors and interactions with students the teacher needs to enact in order to use the program as intended, and the developers' expectations for student behaviors and interactions during instruction. The FOI instruments are intended to provide a foundation for increasing the understanding of which reform-based instructional materials are most effective in improving student learning, and which components of these programs make the greatest contributions to student achievement.

Research Design and Methods: During Year 1, items for the SQ will be created through literature and expert review. These initial items will then be piloted, field tested with 300 students, and revised. Teacher-level instruments will be prepared, and classroom observers will be trained. In Year 2, data will be collected on all student and teacher-level instruments, including student achievement data. In Years 2 and 3, the research team will complete a series of data analyses intended to test for reliability (internal consistency, inter-rater reliability, cross-instrument consistency) and validity (factorial validity, measurement invariance, predictive validity) of these instruments.

Key Measures: All participating teachers will complete two Teacher Instructional Questionnaires for mathematics and science instruction, while a subset of teachers will participate in Teacher Instructional Observation Protocol and Teacher Instructional Log data collection during math and science lessons. Students will complete two SQ's for mathematics and science. Student outcome data, including state standardized test scores in mathematics and science, grades in mathematics and science, and other district/state assessment scores will be collected in all participating classrooms.

Data Analytic Strategy: Data analyses will include tests for reliability (internal consistency, inter-rater reliability, cross-instrument consistency) and validity (factorial validity, measurement invariance, predictive validity). Multiple approaches to instrument validation will be used including: (1) examining reliability across respondents by calculating the extent of inter-rater agreement; (2) examining the internal structure of the data empirically and in relationship to expected results; (3) examining differences in psychometric properties of FOI measures across types of programs (e.g., mathematics and science) and between known groups; (4) examining the agreement between two different sources of information about a program and its operations (e.g., teacher and student reports); and (5) examining the relationship between fidelity measures and expected outcomes for participants.

Project Website: http://researcherswithoutborders.org/projects/measuring-enactment