Project Activities
Research activities will be conducted across multiple phases. In Phase 1, the research team will develop an initial version of the IMFR based on a review of existing measures and feedback from a group of MTSS experts and school personnel. In Phase 2, the measure will undergo iterative administrations, psychometric testing, and refinement. In Phase 3, researchers will conduct psychometric testing on the revised IMFR, including additional validity tests, and examine the cost for intended users.
Structured Abstract
Setting
The project will take place in elementary schools across Michigan, Missouri, Montana, Rhode Island, and Washington.
Sample
The sample will include participants in 100 elementary schools. Within these schools, participants will include 6-8 MTSS team members (typically comprised of administrators; general and special education teachers; and academic and behavior interventionists, specialists, and/or coaches), 20-40 instructional staff not on the MTSS team, and 150-600 students.
Assessment
The IMFR will be a freely available tool that is designed to assess the implementation of integrated MTSS. The tool will measure the degree to which the critical components (and subcomponents) of integrated MTSS are in place (for example, whether they are not implemented, partially implemented, or fully implemented). The critical components are infrastructure (including the MTSS team, leadership, and professional development for staff); assessment (including universal screening and regular progress monitoring); and intervention (including evidence-based interventions and practices across all tiers and increasingly intensive interventions across tiers). Strong implementation will consider the degree to which processes that typically occur separately for academics and behavior are appropriately integrated or aligned. It will include instructions for administration, scoring, a listing of evidence sources that can be used to make rating decisions, and a description of rater requirements. The IMFR will be designed for school teams to use as part of their own monitoring and continuous improvement activities, as well as by external entities engaged in research or evaluation activities.
Research design and methods
The research will be conducted across multiple phases. During Phase 1 (Year 1), the research team will work closely with a technical advisory group comprised of MTSS experts in academics and behavior to develop an initial version of the IMFR. In Phase 2 (Years 2 and 3), the IMFR will be administered in the fall of each year. Two raters will interview relevant MTSS team members and review relevant documentation together and then independently rate each school. Such documentation includes data meeting agendas and minutes as well as documentation of intervention and assessment use. Subsequent psychometric analyses will inform any revisions necessary to yield a reliable and valid measure. During Phase 3 (Years 4 and 5), researchers will administer a revised IMFR, conduct the same psychometric analyses as in Phase 2, and conduct additional analyses to establish the IMFR's concurrent validity, predictive validity, and the validity of the IMFR as a self-assessment. Researchers will also conduct an analysis of the IMFR's cost and usability for the intended users. In Year 5, the research team will make the final IMFR publicly available and disseminate findings to other researchers, district and school administrators, state and local decision-makers, and technical assistance providers.
Control condition
Due to the nature of the research design, there is no control condition.
Key measures
In addition to the assessment being developed (the IMFR), measures include the Usage Rating Profile to assess the perceived utility of the IMFR and a school staff survey to measure perceptions of MTSS implementation among instructional staff who are not a part of the MTSS team. In addition, focus groups will be conducted with MTSS team members and other instructional staff to gather more detailed information on the usability of the IMFR. Schools will provide student data from reading and mathematics tests and the number of in- and out-of-school suspensions. The research team will also collect data on school-level demographics to be used as covariates in the predictive validity analyses.
Data analytic strategy
Multiple techniques will be used to analyze the psychometric properties of the IMFR. As part of Phase 2, the research team will examine the measure's content validity using item-measure correlations and item-fit statistics. For substantive validity, they will examine the psychometric functioning of the response categories. For structural validity, they will conduct a principal components analysis. To establish generalizability, the researchers will conduct differential item functioning analysis, estimate the reliability for each construct, and analyze the presence of rater effects. During Phase 3, the research team will examine the measure's concurrent validity using a multi-trait, multi-method approach to analyze the correlation between scores on the IMFR and school staff perceptions of MTSS implementation. To examine predictive validity, they will use a multilevel model to analyze the correlation between school-level implementation scores and student outcomes. Exploratory analyses will be conducted comparing schools' self-report responses to external ratings to examine whether the IMFR can be completed by schools to help them assess their MTSS implementation progress and needs. Researchers will examine the utility of the IMFR by descriptively analyzing responses on the Usage Rating Profile and conducting a qualitative analysis of focus group data. Finally, the cost of using the tool will be examined using data on the amount of time spent administering and scoring the measure and estimating the proportion of salary related to this time.
People and institutions involved
IES program contact(s)
Products and publications
Products: This project will result in a fully developed and validated measure of MTSS implementation, the IMFR, along with supporting materials to enhance its feasibility and use, such as instructions for administration and scoring and a description of requirements for raters. Products will also include peer-reviewed publications, presentations, reports, and other products for policymaker, practitioner, and researcher audiences.
Supplemental information
Co-Principal Investigators: Lembke, Erica; Riley-Tillman, Chris
Related Network Teams:
- Multi-Tiered Systems of Support Research Network Leadership Team
(PI: Michael Coyne, R324A180020) - Evaluating the Impact of Integrated Behavior and Reading Multi-Tiered Systems of Support in Elementary Schools
(PI: Michael Coyne, R324A190012) - Cohesive Integration of Behavior Support within a Process of Data-Based Intervention Intensification
(PI: Nathan Clemens, R324N180018) - Enhancing Ci3T: Building Professional Capacity for High Fidelity Implementation to Support Students' Educational Outcomes (Project ENHANCE)
(PI: Kathleen Lane, R324N190002)
Questions about this project?
To answer additional questions about this project or provide feedback, please contact the program officer.
