Skip Navigation

Measuring Implementation of the Response to Intervention framework in Milwaukee Public Schools

Region:
Midwest
Description:
The purpose of this study was to determine the reliability of a newly-developed hybrid system to measure the implementation of Response to Intervention (RTI), to determine schools’ progress toward implementing RTI; and to determine whether implementation ratings were related to contextual factors. School improvement coaches were trained and certified to conduct school data reviews. These reviewers visited 70 elementary schools serving grades K-5 in a single urban school district. During each visit, two reviewers made ratings on the 34-indicator rubric and entered their ratings into a dashboard system. Reviewers reconciled discrepant ratings and the reconciled ratings were analyzed. To determine the reliability of the rubric, the study team estimated inter-rater reliability using percent agreement and Cohen’s Kappa to account for chance ratings. Coefficient alphas were calculated to estimate inter-item reliability. To determine how well schools were implementing RTI, average ratings were calculated for each school on the total rubric and six components and converted into categories: “little fidelity”, “inadequate fidelity”, “adequate fidelity”, and “full fidelity”. The study team also calculated Pearson product-moment correlations to study relationships between implementation ratings and characteristics of teachers and students in the schools. Results indicated that the ratings made by the trained data reviewers were reliable even when accounting for chance. Among the 68 visited schools that had complete data, 53 percent of the schools were implementing RTI with adequate fidelity after two years. However, 68 percent of the priority schools did not reach adequate levels of implementation fidelity. Findings also revealed that most schools have yet to implement instruction for diverse students and Tier III instruction with fidelity. Of the contextual factors studied, correlations were found between implementation scores and teacher and student characteristics. The system can be used to produce reliable evidence about the level of RTI implementation in schools and which components of RTI need to be the focus of professional development and coaching. Also, if RTI is indeed an effective school improvement strategy, then by monitoring implementation fidelity of RTI, school districts can improve the chances that RTI produces the expected impacts in their school settings. Establishing an implementation monitoring system requires district staff time to complete training, conduct the data reviews, resolve rating discrepancies, and enter the data into a dashboard system.
Publication Type:
What's Happening
Online Availability:
Publication Date:
November 2016
Contact: