Description:

Previous Award Number: R305A170284
Previous Awardee: University of Missouri

Co-Principal Investigators: Eklund, Katie; von der Embse, Nathaniel

Purpose: The purpose of this project was to support the development and validation of the Intervention Selection Profile (ISP), a suite of brief assessment tools to identify appropriate interventions for students in need of Tier 2 support. In a response to intervention (RTI) framework, Tier 2 interventions are for students who are struggling to make adequate progress in response to general classroom-level (Tier 1) instruction and supports. A large body of research has supported the effectiveness of several targeted Tier 2 behavioral interventions, such as Check In/Check Out and social skills instruction, yet additional research suggests that these supports may be more effective if they are tailored to address individual student needs. For students in need of Tier 2 support, the ISP allows educators to quickly and easily collect problem-identification data indicative of each student's needs, including the function of problem behaviors and deficits in positive skills, and to select and adapt appropriate interventions in response.

Project Activities: This project had several aims that built on one another to create a solid evidence framework for the ISP and its use. The research team developed and refined each ISP assessment tool and the decision-making process by which ISP data are evaluated to identify why students engage in problem behavior or demonstrate social skills deficits. Each measure was compared to functional behavior assessment through systematic direct observations and skill assessment rating scales such as the Social Skills Improvement System. The researchers identified the most efficient yet effective approach to training teachers to collect ISP data and use the data to inform intervention adaptation. Finally, the researchers conducted single-case design studies to evaluate the treatment utility of the ISP tools to examine the extent to which each tool contributed to positive intervention outcomes.

Key Outcomes: The main features of the assessment tools and outcomes of the validation study are as follows:

• The Intervention Selection Profile consists of two assessments (ISP-Skills and ISP-Function) and training materials for educators.
• ISP-Skills is a valid and reliable 14-item measure of student social-emotional functioning to inform the delivery of Tier 2 interventions (Kilgus et al., 2020)
• ISP-Function is a valid and reliable 5-item measure of the consequences of maintaining problem behaviors to inform the selection of function-based Tier 2 interventions (Eklund et al., 2023; Kilgus et al., 2019)
• ISP automated training protocols instruct educators on how to collect ISP data and use the data to inform intervention decisions (Kilgus et al., in press; von der Embse et al., 2022; Zahn et al., 2023).
• An open-access web-based scoring system for educators to complete and score the ISP-Skills. (https://smhcollaborative.org/isp-materials/)

Structured Abstract

Setting: Students and teachers for this study were recruited from Missouri and Florida.

Sample: For the development work, the team recruited 170 teachers who provided ratings for a targeted 850 students. The team also recruited 60 kindergarten through grade 5 (K–5) students and teachers to administer some of the alternate criterion measures for validation purposes. For the training protocol study, the team recruited 160 teachers from each of the 2 research sites. For the single-case design studies, the team recruited 11 K–5 students and their teachers across the 2 sites.

Assessment: The ISP suite of tools comprise two measures: ISP-Skills and ISP-Function. ISP-Skills (14 items) supports adaptation of instructional interventions to student needs via the detection of acquisition deficits within three broad skill domains: social skills, academic enablers, and emotional competence. Each of these domains is measured by a unique subscale within ISP-Skills. ISP-Function is a brief functional assessment tool (5 items) intended to support adaptation of contingency management interventions to student needs via determination of the function of certain problem behaviors. Users, such as general education teachers, provide a brief rating to represent the extent to which they observe students engaging in various problem behaviors during a pre-specified period, as well as the frequency with which such behaviors are met with four types of consequences: adult attention, peer attention, escape/avoidance, and access to tangibles or activities. Both measures are similar to, but briefer than, existing problem identification tools.

Research Design and Methods: The research team completed a sequence of related studies. First, the research team developed and refined the two ISP tools, as well as the decision-making structures upon which they are founded. The team also established initial evidence for the validity of the ISP tools. Next, the team conducted studies to identify the most efficient yet effective approach to training teachers to collect and use ISP data. They examined varying approaches to teacher training, including basic exposure, as well as training with practice and performance feedback. In the final phase of this project, through single-case design studies, the team evaluated the treatment utility of ISP tools and the broader ISP procedural framework. Of specific interest was whether the assessment process affects key indicators of student behavioral functioning (e.g., attendance, suspensions) and academic performance (e.g., benchmark scores, statewide achievement test performance).

Control Condition: There was no control condition for this measurement study.

Key Measures: In addition to the ISP itself, the researchers used six other measures designed to assess student skills or evaluate functional behavior, and they used information from these measures to validate and refine the ISP's content and administration. These other measures include the Social Skills Improvement System (SSIS), the Academic Competence Evaluation Scales (ACES), the Devereux Student Strengths Assessment (DESSA), the Functional Assessment Checklist for Teachers and Staff (FACTS), the Functional Analysis Screening Tool (FAST), and systematic direct observation by teachers or counselors.

Data Analytic Strategy: For the development work, the team calibrated and scaled the ISP using item response theory and diagnostic cognitive modeling. Analyses to support initial validation of the ISP included structural analyses, correlational analyses between the ISP and other similar instruments, reliability analyses (internal consistency and test-retest), diagnostic accuracy, and analyses of test fairness (e.g., differential item functioning). For the training study, the researchers calculated accuracy scores that represent the difference between ISP scores and a true score derived through systematic direct observation. These accuracy scores were used as the outcome in repeated measures MANOVA tests and for chi-square goodness-of-fit tests to examine how many individuals within each training group generated accurate decisions with the ISP. The researchers analyzed systematic direct observation data from the single-case design studies using Koehler and Levin's regulated randomization test.

Related IES Projects: Project MIDAS: Development of a Multi-Informant Decisional Assessment System (R305A210019)

Products and Publications

ERIC Citations: Find available citations in ERIC for this award here or here.

Project website: https://smhcollaborative.org/isp

Select Publications:

Eklund, K., Kilgus, S. P., Zahn, M. R., von der Embse, N. P., Willenbrink, J., Davis, E., & Twombly, T. (2023). Intervention Selection Profile-Function: An examination of decisional accuracy relative to traditional FBA data. School Psychology, 38(2), 110–118. doi.org/10.1037/spq0000524

Kilgus, S. P., Bonifay, W. E., Eklund, K., Nathaniel, P., Peet, C., Izumi, J., ... & Meyer, L. N. (2020). Development and validation of the Intervention Skills Profile–Skills: A brief measure of student social-emotional and academic enabling skills. Journal of School Psychology, 83, 66–88.

Kilgus, S. P., Eklund, K., & von der Embse, N. P. (2019). Psychometric defensibility of the Intervention Selection Profile-Social Skills (ISP-SS) with students at risk for behavioral concerns. Psychology in the Schools, 56(4), 526–538.

Kilgus, S. P., von der Embse, N. P., Eklund, K., Izumi, J., Peet, C., Meyer, L., & Taylor, C. N. (2019). Reliability, validity, and accuracy of the intervention selection profile—Function: A brief functional assessment tool. School Psychology, 34(5), 531.

von der Embse, N. P., Kilgus, S. P., Eklund, K., Zahn, M., Peet, C., & Durango, S. (2021). Promoting effective decision-making: Training educators to collect and use social-emotional skill assessment data to inform Tier II interventions. School Psychology Review. Advance online publication. doi.org/10.1080/2372966X.2020.1827680