Co-Principal Investigator: Cook, Clayton R.

Purpose: The purpose of this project is to further refine and evaluate the Treatment Integrity Instrument of Elementary Settings, consisting of teacher- (TIES-T) and observer-report (TIES-O) versions, for use in a quality improvement (QI) process. The QI process is designed to support the implementation and sustainment of universal evidence-based programs (EBPs) for student social, emotional, and behavioral (SEB) needs in elementary schools. The researchers will iteratively refine and validate the TIES-T (initially developed in a prior IES Development and Innovation project) to be a pragmatic, technically sound, and change sensitive self-report measure for use within a locally-managed QI process to improve both EBP implementation and student SEB outcomes.

Project Activities: In this project, the research team will iteratively refine the TIES-T to establish content validity, customizability, and usability. They will also examine methods to maximize the accuracy of the TIES-T. They will evaluate the TIES-T score reliability, validity, and change sensitivity and assess the ability of the TIES-T to drive EBP implementation and sustainment.

Products: The primary product of this project will be the Treatment Integrity Instrument of Elementary Settings — Teachers (TIES-T). TIES-T will capture the quantity (adherence) and quality (competence) of teachers' delivery of practice elements found in universal social, emotional, and behavioral interventions in elementary schools.

Structured Abstract

Setting: The team will begin to refine the TIES-T in laboratory settings before being they further refine and validate it in elementary schools in Virginia and then in Minnesota.

Sample: Over four years, iterative tests of the TIES-T will include approximately 300 staff sampled from 20 schools in Virginia and Minnesota to be demographically representative of urban and suburban schools.

Assessment: The TIES-T will be a technically sound, feasible, appropriate, and acceptable self-report treatment integrity measure that captures key dimensions related to the high-quality delivery of practice elements that comprise effective universal evidenced-based practices (EBPs) targeting student SEB functioning. It will be agnostic with respect to specific EBPs by including items that can be aligned with the identified core practice elements of a given EBP and characterize the practices used by teachers in business as usual. The final product will be a suite of treatment integrity measures, the TIES, comprised of parallel self- (TIES-T) and observer-report (TIES-O) treatment integrity measures designed to support the uptake, use, and sustainment of universal SEB EBPs in elementary schools.

Research Design and Methods: The researchers will complete a series of five studies within an iterative process to guide refinement of the TIES-T. Study 1 will focus on refinement of the TIES-T with review and feedback from research and practice experts evaluating item coverage and breadth. Study 2 will include usability testing with end users (administrators, coaches, teachers) focused on the feasibility, customizability, and usability of the TIES-T. Study 3 will involve an experimental study intended to inform efforts to optimize the accuracy of the TIES-T relative to the TIES-O. Study 4 will include data collection with schools at multiple time points to examine preliminary convergent, divergent, discriminant, and predictive validity. Study 5 will evaluate the usability of the data reports from the TIES-T to inform efforts to use the data to promote behavior change.

Control Condition: Due to the nature of this project, there is no control condition.

Key Measures: Though the primary measure is the one proposed for development, the TIES-T, other measures include the TIES-O, Educator Self-Efficacy Scale, Evidence-Based Attitudes Scale, the Usability Rating Profile, education outcomes (GPA, standardized test scores), classroom-level behavioral outcomes (academic engaged time, disruptive behavior, prosocial behavior), and costs.

Data Analytic Strategy: The research team will use mixed-method qualitative and quantitative data analyses for studies 1-3 and 5 to inform key revisions to the TIES-T to improve its reliability, validity, and usability. In study 4 they will use a multi-trait multi-method quantitative framework to examine evidence of convergent and divergent validity and item response theory to examine whether specific or broad items most accurately capture key dimensions of teachers' delivery of common practice elements of school-wide positive behavioral interventions and supports with integrity.

Cost Analysis: The team will analyze costs of TIES-T training and administration using the "ingredients approach." The first two steps of the ingredients approach are to (1) measure all activities and resources ("ingredients") involved in an activity (e.g., measure completion, scoring) and (2) assign cost values to each measured ingredient. Specifically, they will use the CostOut program to complete the cost analysis for TIES-T training and administration. CostOut allows users to list the ingredients measured, assign prices (national and user-inputted local values), and calculate direct and indirect costs based on the units per ingredient used. Using CostOut will thus allow us to estimate the total costs for TIES-T training and administration.

Related IES Projects: Development and Validation of a Treatment Integrity Measure of Classroom-Based Instructional Interventions in Early Childhood Settings (R305A140487); BEST in CLASS-Elementary: A Preventative Classroom-based Intervention Model (R305A150246); Promoting Social, Emotional, and Behavioral Competence in Young High-Risk Children: A Preventative Classroom-Based Early Intervention Model (R324A080074), Efficacy of the BEST in CLASS Intervention for Young Children at High Risk for Emotional and Behavioral Disorders (R324A110173), BEST in CLASS-Web: A Web-Based Intervention Supporting Early Childhood Teachers' Use of Evidence-Based Practices with Young Children at Risk for Emotional/Behavioral Disorders (R324A160158), A Conceptual Replication of BEST in CLASS: An Efficacy Study of BEST in CLASS-Elementary (R305A180182)