Project Activities
In this project, the research team will iteratively refine the TIES-T to establish content validity, customizability, and usability. They will also examine methods to maximize the accuracy of the TIES-T. They will evaluate the TIES-T score reliability, validity, and change sensitivity and assess the ability of the TIES-T to drive EBP implementation and sustainment.
Structured Abstract
Setting
The team will begin to refine the TIES-T in laboratory settings before being they further refine and validate it in elementary schools in Virginia and then in Minnesota.
Sample
Over four years, iterative tests of the TIES-T will include approximately 300 staff sampled from 20 schools in Virginia and Minnesota to be demographically representative of urban and suburban schools.
The TIES-T will be a technically sound, feasible, appropriate, and acceptable self-report treatment integrity measure that captures key dimensions related to the high-quality delivery of practice elements that comprise effective universal evidenced-based practices (EBPs) targeting student SEB functioning. It will be agnostic with respect to specific EBPs by including items that can be aligned with the identified core practice elements of a given EBP and characterize the practices used by teachers in business as usual. The final product will be a suite of treatment integrity measures, the TIES, comprised of parallel self- (TIES-T) and observer-report (TIES-O) treatment integrity measures designed to support the uptake, use, and sustainment of universal SEB EBPs in elementary schools.
Research design and methods
The researchers will complete a series of five studies within an iterative process to guide refinement of the TIES-T. Study 1 will focus on refinement of the TIES-T with review and feedback from research and practice experts evaluating item coverage and breadth. Study 2 will include usability testing with end users (administrators, coaches, teachers) focused on the feasibility, customizability, and usability of the TIES-T. Study 3 will involve an experimental study intended to inform efforts to optimize the accuracy of the TIES-T relative to the TIES-O. Study 4 will include data collection with schools at multiple time points to examine preliminary convergent, divergent, discriminant, and predictive validity. Study 5 will evaluate the usability of the data reports from the TIES-T to inform efforts to use the data to promote behavior change.
Control condition
Due to the nature of this project, there is no control condition.
Key measures
Though the primary measure is the one proposed for development, the TIES-T, other measures include the TIES-O, Educator Self-Efficacy Scale, Evidence-Based Attitudes Scale, the Usability Rating Profile, education outcomes (GPA, standardized test scores), classroom-level behavioral outcomes (academic engaged time, disruptive behavior, prosocial behavior), and costs.
Data analytic strategy
The research team will use mixed-method qualitative and quantitative data analyses for studies 1-3 and 5 to inform key revisions to the TIES-T to improve its reliability, validity, and usability. In study 4 they will use a multi-trait multi-method quantitative framework to examine evidence of convergent and divergent validity and item response theory to examine whether specific or broad items most accurately capture key dimensions of teachers' delivery of common practice elements of school-wide positive behavioral interventions and supports with integrity.
Cost analysis strategy
The team will analyze costs of TIES-T training and administration using the "ingredients approach." The first two steps of the ingredients approach are to (1) measure all activities and resources ("ingredients") involved in an activity (e.g., measure completion, scoring) and (2) assign cost values to each measured ingredient. Specifically, they will use the CostOut program to complete the cost analysis for TIES-T training and administration. CostOut allows users to list the ingredients measured, assign prices (national and user-inputted local values), and calculate direct and indirect costs based on the units per ingredient used. Using CostOut will thus allow us to estimate the total costs for TIES-T training and administration.
People and institutions involved
IES program contact(s)
Project contributors
Products and publications
The primary product of this project will be the Treatment Integrity Instrument of Elementary Settings - Teachers (TIES-T). TIES-T will capture the quantity (adherence) and quality (competence) of teachers' delivery of practice elements found in universal social, emotional, and behavioral interventions in elementary schools.
Related projects
Questions about this project?
To answer additional questions about this project or provide feedback, please contact the program officer.
