Co-Principal Investigators: Lembke, Erica; Riley-Tillman, Chris

Purpose: The purpose of this study is to develop and validate the Integrated MTSS Fidelity Rubric (IMFR), a tool for measuring school-level implementation of integrated multi-tiered systems of support (MTSS). Integrated MTSS provides multiple levels of coordinated supports to meet the academic and behavioral needs of all learners. Although MTSS is being planned and implemented in schools around the country, there are several challenges related to the integration of academic (such as math and reading) and behavioral supports and limited research on what is needed to implement integrated MTSS in a way that leads to improved student outcomes. The combination of the widespread use of MTSS and the complexities and challenges of implementation (for instance, the difficulty assessing student progress and adapting instruction and the tendency for students with disabilities to be served outside of MTSS) highlights the need for a tool that reliably and validly measures implementation of integrated MTSS and can be used for multiple purposes, including by schools and districts to guide implementation and by researchers to advance our understanding of the impacts of MTSS on student outcomes. While tools for assessing MTSS implementation exist, they are either limited in terms of their content, usability, or psychometric evidence or the degree to which they address the integration of academic and behavioral supports. The current study will address these limitations by developing and validating a tool to assess the implementation fidelity of integrated MTSS, examining the relationship between the implementation of the essential components of MTSS (as measured by the tool) and student outcomes, and examining the cost and perceived utility of implementing and scoring the assessment.

Project Activities: Research activities will be conducted across multiple phases. In Phase 1, the research team will develop an initial version of the IMFR based on a review of existing measures and feedback from a group of MTSS experts and school personnel. In Phase 2, the measure will undergo iterative administrations, psychometric testing, and refinement. In Phase 3, researchers will conduct psychometric testing on the revised IMFR, including additional validity tests, and examine the cost for intended users.

Products: This project will result in a fully developed and validated measure of MTSS implementation, the IMFR, along with supporting materials to enhance its feasibility and use, such as instructions for administration and scoring and a description of requirements for raters. Products will also include peer-reviewed publications, presentations, reports, and other products for policymaker, practitioner, and researcher audiences.

Structured Abstract

Setting: The project will take place in elementary schools across Michigan, Missouri, Montana, Rhode Island, and Washington.

Sample: The sample will include participants in 100 elementary schools. Within these schools, participants will include 6-8 MTSS team members (typically comprised of administrators; general and special education teachers; and academic and behavior interventionists, specialists, and/or coaches), 20-40 instructional staff not on the MTSS team, and 150-600 students.

Assessment: The IMFR will be a freely available tool that is designed to assess the implementation of integrated MTSS. The tool will measure the degree to which the critical components (and subcomponents) of integrated MTSS are in place (for example, whether they are not implemented, partially implemented, or fully implemented). The critical components are infrastructure (including the MTSS team, leadership, and professional development for staff); assessment (including universal screening and regular progress monitoring); and intervention (including evidence-based interventions and practices across all tiers and increasingly intensive interventions across tiers). Strong implementation will consider the degree to which processes that typically occur separately for academics and behavior are appropriately integrated or aligned. It will include instructions for administration, scoring, a listing of evidence sources that can be used to make rating decisions, and a description of rater requirements. The IMFR will be designed for school teams to use as part of their own monitoring and continuous improvement activities, as well as by external entities engaged in research or evaluation activities.

Research Design and Methods: The research will be conducted across multiple phases. During Phase 1 (Year 1), the research team will work closely with a technical advisory group comprised of MTSS experts in academics and behavior to develop an initial version of the IMFR. In Phase 2 (Years 2 and 3), the IMFR will be administered in the fall of each year. Two raters will interview relevant MTSS team members and review relevant documentation together and then independently rate each school. Such documentation includes data meeting agendas and minutes as well as documentation of intervention and assessment use. Subsequent psychometric analyses will inform any revisions necessary to yield a reliable and valid measure. During Phase 3 (Years 4 and 5), researchers will administer a revised IMFR, conduct the same psychometric analyses as in Phase 2, and conduct additional analyses to establish the IMFR's concurrent validity, predictive validity, and the validity of the IMFR as a self-assessment. Researchers will also conduct an analysis of the IMFR's cost and usability for the intended users. In Year 5, the research team will make the final IMFR publicly available and disseminate findings to other researchers, district and school administrators, state and local decision-makers, and technical assistance providers.

Control Condition: Due to the nature of the research design, there is no control condition.

Key Measures: In addition to the assessment being developed (the IMFR), measures include the Usage Rating Profile to assess the perceived utility of the IMFR and a school staff survey to measure perceptions of MTSS implementation among instructional staff who are not a part of the MTSS team. In addition, focus groups will be conducted with MTSS team members and other instructional staff to gather more detailed information on the usability of the IMFR. Schools will provide student data from reading and mathematics tests and the number of in- and out-of-school suspensions. The research team will also collect data on school-level demographics to be used as covariates in the predictive validity analyses.

Data Analytic Strategy: Multiple techniques will be used to analyze the psychometric properties of the IMFR. As part of Phase 2, the research team will examine the measure's content validity using item-measure correlations and item-fit statistics. For substantive validity, they will examine the psychometric functioning of the response categories. For structural validity, they will conduct a principal components analysis. To establish generalizability, the researchers will conduct differential item functioning analysis, estimate the reliability for each construct, and analyze the presence of rater effects. During Phase 3, the research team will examine the measure's concurrent validity using a multi-trait, multi-method approach to analyze the correlation between scores on the IMFR and school staff perceptions of MTSS implementation. To examine predictive validity, they will use a multilevel model to analyze the correlation between school-level implementation scores and student outcomes. Exploratory analyses will be conducted comparing schools' self-report responses to external ratings to examine whether the IMFR can be completed by schools to help them assess their MTSS implementation progress and needs. Researchers will examine the utility of the IMFR by descriptively analyzing responses on the Usage Rating Profile and conducting a qualitative analysis of focus group data. Finally, the cost of using the tool will be examined using data on the amount of time spent administering and scoring the measure and estimating the proportion of salary related to this time.

Related Network Teams:

• Multi-Tiered Systems of Support Research Network Leadership Team
  (PI: Michael Coyne, R324A180020)
• Evaluating the Impact of Integrated Behavior and Reading Multi-Tiered Systems of Support in Elementary Schools
  (PI: Michael Coyne, R324A190012)
• Cohesive Integration of Behavior Support within a Process of Data-Based Intervention Intensification
  (PI: Nathan Clemens, R324N180018)
• Enhancing Ci3T: Building Professional Capacity for High Fidelity Implementation to Support Students' Educational Outcomes (Project ENHANCE)
  (PI: Kathleen Lane, R324N190002)