Project Activities
Research activities were conducted across multiple phases. In phase 1, the research team developed an initial version of the IMFR based on a review of existing measures and feedback from a group of MTSS experts and school personnel. In phase 2, the measure underwent iterative administrations, psychometric testing, and refinement. In phase 3, researchers conducted a final round of psychometric testing on the revised IMFR and examined the IMFR’s cost and utility for intended users. Although some dissemination activities, such as manuscript submissions, occurred in the final years of the study, other dissemination activities (e.g., conference presentations) occurred throughout the life of the study.
Structured Abstract
Setting
The project took place in elementary schools in nine states located in the United States.
Sample
The sample included 65-87 schools (across 13-20 districts) each year across the 3 years of data collection. Within these schools, participants included MTSS leadership team members (typically comprised of administrators; general and special education teachers; and academic and behavior interventionists, specialists, and/or coaches). Seventeen of these schools also participated in the cost study.
The final version of the IMFR is a freely available tool that assesses the implementation of integrated MTSS. The tool measures the degree to which academic and social-emotional behavioral (SEB) supports are strategically combined (integrated) in the core components of I-MTSS: Instruction and Intervention, Assessment, Data-based Decision Making, and Infrastructure. The tool is a part of a suite of materials that includes (1) a self-paced online training series on how to administer and score the IMFR, (2) a materials packet that contains the rubric itself, a scripted interview protocol, and a scoring guidance document, and (3) a user manual. The IMFR is designed for school teams to use as part of their own monitoring and continuous improvement activities, as well as by external entities engaged in research or evaluation activities.
Research design and methods
The research was conducted across multiple phases. During phase 1, the research team worked closely with a technical advisory group comprised of MTSS experts in academics and behavior to develop an initial version of the IMFR. In phases 2 and 3, the IMFR was administered in schools. Data were analyzed to better understand the tool’s content, substantive, and structural validity and inter-rater reliability. These results were used to iteratively refine the measure. Phase 3 also included additional exploratory analyses to explore the IMFR's concurrent validity and predictive validity and an analysis of the IMFR's cost and usability for the intended users. The IMFR was made publicly available in March 2024.
Control condition
Due to the nature of the research design, there was no control condition.
Key measures
In addition to the assessment being developed (i.e., the IMFR), focus groups were conducted with MTSS team members to gather more detailed information on the usability of the IMFR. The research team also collected data on school-level demographics.
Data analytic strategy
Multiple techniques were used to analyze the psychometric properties of the IMFR. The research team examined the measure's content validity using item-measure correlations, item-fit statistics, and item difficulty statistics. For substantive validity, they examined the number of responses per scale category, the ordered average measure per category, ordered category thresholds, and unweighted category mean square statistics of less than 2.0 for each scale category. For structural validity, they conducted exploratory and confirmatory factor analyses. To establish generalizability, the researchers analyzed the presence of rater effects and the reliability of each construct. The study team examined the measure's concurrent validity with the Tiered Fidelity Inventory (TFI) with a subset of 42 schools using a multi-trait, multi-method approach. To explore predictive validity, they used a regression model to analyze the correlation between IMFR results and student outcomes as reported in publicly available school report card data. Researchers examined the utility and costs of the IMFR conducting a qualitative analysis of focus group data.
Cost analysis strategy
The research team conducted a cost analysis during phase 3 of the project.
Key outcomes
The main findings of this project, as reported by the principal investigator, are as follows:
- Psychometric results indicate that the IMFR is a valid and reliable measure of elementary school I-MTSS.
- With respect to content validity, the domains and items accurately reflect I-MTSS. For structural validity, the item-level ratings accurately reflect a school’s I-MTSS implementation status.
- Reliability results were also strong, indicating that multiple raters produce similar ratings for schools, although some raters are more severe than others.
- Social validity results indicate that the benefits of using the IMFR outweigh the costs, the IMFR is valuable to schools, and schools would recommend others use it. However, for substantive validity, scale categories may need refinement in future testing.
- Based on usability testing, 70% of the teams, or 12 of 17, reported that the benefits of participating in the IMFR administration outweighed the costs.
People and institutions involved
IES program contact(s)
Project contributors
Products and publications
Publications:
ERIC Citations: Find available citations in ERIC for this award here.
Select Publications:
Majeika, C. E., Pierce, J., Smith, H., Lembke, E., & Gandhi, A. (2024). Integrated Multi-Tiered Systems of Support in Elementary Schools: Practical Applications. Intervention in School and Clinic, Intervention in School and Clinic, 60(1), 53-61.
Related projects
Questions about this project?
To answer additional questions about this project or provide feedback, please contact the program officer.
