Co-Principal Investigators: Kate Eklund; Nathan von der Embse (University of South Florida)

Previous Grant Number: R305A170284
Previous University Affiliation: University of Missouri, Columbia

Purpose: The overarching purpose of this project is to support the development and validation of the Intervention Selection Profile (ISP), a suite of brief problem identification tools. Within the Response to Intervention (RTI) framework, Tier 2 interventions are those that are targeted to students who are struggling to make adequate progress in response to general classroom-level (Tier 1) instruction and supports. A large body of research has supported the effectiveness of several targeted Tier 2 behavioral interventions, such as Check In/Check Out and social skills instruction, yet additional research suggests that these supports may be more effective if they are tailored to address individual student needs. For students in need of Tier 2 support, the ISP will allow educators to quickly and easily collect problem identification data indicative of each student's concerns, including the function of problem behaviors and deficits in positive skills, and select and adapt appropriate interventions in response.

Project Activities: This project has several aims that build on one another to create a solid evidence framework for the ISP and its use. The research team will support the development and refinement of (a) each ISP tool, including the items upon which each measure is founded, and (b) the decision making process by which ISP data are evaluated to identify why students engage in problem behavior or demonstrate acquisition deficits. Each measure will be compared to multiple criterion outcome measures, including gold standard functional behavior assessment instruments (e.g., systematic direct observations) and skill assessment rating scales (e.g., Social Skills Improvement System). Secondly, researchers will identify the most efficient yet effective approach to training teachers to collect ISP data and use it to inform intervention adaptation. Studies will examine varying approaches to teacher training to determine the most feasible means to promote collection and use of ISP data. A third goal is to examine the treatment utility of each ISP tool. The researchers will conduct single-case design studies to examine the extent to which each tool contributes to positive intervention outcomes. Finally, researchers will conduct a randomized controlled trial (RCT) to evaluate the treatment utility of the broader ISP procedural framework, wherein ISP tools are paired with universal screeners to inform Tier 2 intervention.

Products: The researchers will produce a fully developed and validated version of the ISP instrument, and peer-reviewed publications.

Structured Abstract

Setting: Students and teachers for this study will be recruited from Missouri and Florida.

Sample: For the Year 1 development work, the team will recruit approximately 170 teachers who will provide ratings for a targeted 850 students. The team will also seek to recruit 60 K–5th grade students and teachers in Year 1 to administer some of the alternate criterion measures for validation purposes. For the training protocol study in Year 2, the team will recruit approximately 160 teachers from each of the two research sites. For the single-case design studies in Year 3, the team will recruit approximately 20 K–5th grade students and their teachers across the two sites. For the Year 4 study, the team will recruit 60 teachers and approximately 180 students to validate the overall ISP framework.

Assessment: The ISP suite of tools will comprise two measures: ISP-Skills and ISP-Function. The ISP-Skills supports adaptation of instructional interventions to student needs via the detection of acquisition deficits within three broad skill domains: Social Skills, Academic Enablers, and Emotional Competence. Each of these domains is measured by a unique subscale within the ISP-Skills. The ISP-Function is a brief functional assessment tool (5 items), intended to support adaptation of contingency management interventions to student needs via determination of the function of certain problem behaviors. Users, such as general education teachers, provide a brief rating to represent the extent to which they observe students engaging in various problem behaviors during a pre-specified period, as well as the frequency with which such behaviors are met with four types of consequences: adult attention, peer attention, escape/avoidance, and access to tangibles or activities. Both measures are intended to be similar to but briefer than existing problem identification tools.

Research Design and Methods: The research team will complete a four-year sequence of related studies, all of which will be conducted in K–5 school settings. Studies move from development and refinement activities, to validation, to training and support, to treatment utility of individual ISP instruments and the overall ISP procedural framework. In the first year, the research team will develop and refine the two ISP tools, as well as the decision making structures upon which they will be founded. The team will also establish initial evidence for the validity of the ISP tools. In Year 2, the team will conduct studies to identify the most efficient yet effective approach to training teachers to collect and use ISP data. Studies will examine varying approaches to teacher training, including basic exposure, as well as training with practice and performance feedback. In Years 3 and 4, through single-case design studies and a RCT, the team will evaluate the treatment utility of ISP tools and the broader ISP procedural framework. Of specific interest will be whether the assessment process affects key indicators of student behavioral functioning (e.g., attendance, suspensions) and academic performance (e.g., benchmark scores, statewide achievement test performance).

Control Condition: There is no specific control condition for this measurement study, though the researchers' baseline hypotheses are that the ISP will be faster, easier, and more useful for teachers than other comparable instruments, and that students will respond better to tailored Tier 2 interventions that are assigned via the ISP than they would to other generic Tier 2 supports.

Key Measures: In addition to the ISP itself, the researchers will use six other measures designed to assess student skills or evaluate functional behavior; information from these measures will be used to validate and refine the ISP's content and administration. These other measures include the Social Skills Improvement System (SSIS), the Academic Competence Evaluation Scales (ACES), the Devereux Student Strengths Assessment (DESSA), the Functional Assessment Checklist for Teachers and Staff (FACTS); the Functional Analysis Screening Tool (FAST), and systematic direct observation by teachers or counselors.

Data Analytic Strategy: For the development work in Year 1, the team will calibrate and scale the ISP using item response theory and diagnostic cognitive modeling. Analyses to support initial validation of the ISP will include structural analyses, correlational analyses between the ISP and other similar instruments, reliability analyses (internal consistency and test-retest), diagnostic accuracy, and analyses of test fairness (e.g., differential item functioning). For the administration research in Year 2, the researchers will calculate accuracy scores that represent the difference between ISP scores and a true score derived through systematic direct observation. These accuracy scores will be used as the outcome in repeated measures MANOVA tests, and for chi-square goodness-of-fit tests to examine how many individuals within each training group generated accurate decisions with the ISP. In Year 3, systematic direct observation data from the single-case design studies will be analyzed using Koehler and Levin's regulated randomization test. For the framework validation work in Year 4, the team will analyze data using two-level hierarchical linear models, with students (Level 1) nested within classrooms (Level 2).

Products

ERIC Citations: Find available citations in ERIC for this award here or here.

Select Publications

Journal articles

Kilgus, S. P., Bonifay, W. E., Eklund, K., Nathaniel, P., Peet, C., Izumi, J., ... & Meyer, L. N. (2020). Development and validation of the Intervention Skills Profile–Skills: A brief measure of student social-emotional and academic enabling skills. Journal of School Psychology, 83, 66–88.

Kilgus, S. P., Eklund, K., & von der Embse, N. P. (2019). Psychometric defensibility of the Intervention Selection Profile-Social Skills (ISP-SS) with students at risk for behavioral concerns. Psychology in the Schools, 56(4), 526–538.

Kilgus, S. P., von der Embse, N. P., Eklund, K., Izumi, J., Peet, C., Meyer, L., & Taylor, C. N. (2019). Reliability, validity, and accuracy of the intervention selection profile—Function: A brief functional assessment tool. School Psychology, 34(5), 531. Full text

von der Embse, N. P., Kilgus, S. P., Eklund, K., Zahn, M., Peet, C., & Durango, S. (2021). Promoting effective decision-making: Training educators to collect and use social-emotional skill assessment data to inform Tier II interventions. School Psychology Review. Advance online publication. doi.org/10.1080/2372966X.2020.1827680