Development and Validation of a Treatment Integrity Measure of Classroom-Based Instructional Interventions in Early Childhood Settings
Co-Principal Investigator: Kevin Sutherland
Purpose: Many preschool children exhibit problem behaviors and pre-academic deficits that place them at risk for emotional and behavioral disorders. Evidence-based instructional practicesdelivered by teachers in classrooms have demonstrated positive effects for young children with problem behavior. Efforts to implement and evaluate the evidence-based programs (EBPs) for preschool children with problem behaviors across diverse early childhood classrooms face a number of implementation and evaluation barriers. One of these barriers is the lack of treatment integrity measures designed to assess implementation of EBPs in early childhood classrooms. The purpose of this project is to develop observational and teacher self-report treatment integrity measures designed to characterize the implementation of EBPs in early childhood classrooms.
Project Activities: The project will be carried out in three phases and researchers will incorporate data collected from classrooms participating in a previously funded IES Goal 3 grant, a randomized controlled trial evaluating the efficacy of the BEST in CLASS (BiC) early childhood program, to evaluate the initial psychometrics of the observational and teacher self-report integrity measures. In Phase I, the research team will develop and refine the treatment integrity measures, establishing score reliability and validity (content, construct). During Phase II, researchers will evaluate the generalizability and sensitivity of the integrity measures. The team will evaluate the utility of the observational measure during Phase III.
Products: The products of this project will include fully developed and validated observational and teacher self-report treatment integrity measures designed to characterize the implementation of EBPs delivered by teachers in early childhood classrooms. Peer reviewed publications will also be produced.
Setting: Data will be drawn from the one hundred and twenty early childhood classrooms across urban cities in Virginia which are participating in an existing BiC efficacy trial are included.
Sample: The Phase I sample will include all children enrolled in the VA site during year 4 of the BiC Efficacy Trial, including approximately 60 children (n=30 BiC; n=30 business as usual) and 20 teachers (n=10 BiC; n=10 business as usual). The Phase II sample will include approximately 120 children with problem behavior (n=60 BiC; n=60 business as usual) and 40 teachers (n=20 BiC; n=20 business as usual). The Phase III sample will include 72 children and 72 teachers.
Assessment: The assessment system that will be created will evaluate three components of treatment integrity: (a) Treatment adherence—the extent to which a teacher delivers instructional practices “prescribed” by a EBP; (b) Treatment differentiation—the extent to whether (and “to where”) teachers deviate from a treatment protocol, and (c) Competence—the level of skill shown by the teachers in delivering the EBP. The system includes The Treatment Integrity Measure for Early Childhood Settings (TIMECS) which will assess the integrity of instructional practices delivered by teachers in early childhood classrooms. TIMECS will contain 3 scales: Adherence, Competence, and Differentiation. The Treatment Integrity Measure for Early Childhood Settings—Teacher Report (TIMECS-TR) will be developed to assess teacher-report of adherence and differentiation of instructional practices. TIMECS-TR will contain 2 scales: Adherence and Differentiation.
Intervention: BiC is a classroom-based program designed to ameliorate the problem behavior and increase the preacademic learning of young children with problem behavior. BiC is designed to increase the quantity and quality of evidence-based instructional practices implemented by teachers in early childhood classrooms. BiC comprises 10 instructional practices that are empirically linked to positive child behavioral and pre-academic outcomes. First Step to Success is an EBP for young children with problem behavior that includes school and home based components. The Teaching Pyramid intervention is a multi-tiered classroom-based EBP designed to promote social-emotional development and prevent challenging behavior of young children with problem behavior.
Research Design and Methods: Data from classrooms participating in a previously funded IES Goal 3 grant, a randomized controlled trial evaluating the efficacy of the BiC early childhood program, will be used to evaluate the initial psychometrics of the observational and teacher self-report integrity measures. For Phase I and II, real-time classroom observations of children with problem behaviors will be coded using the observational treatment integrity measure; teachers will self-report on their use of instructional practices. Phase III will use videotaped recordings from children and teachers delivering 3 EBPs (Best in Class; First Step to Success; Teaching Pyramid) and a business and usual condition, which will be coded for integrity (adherence, differentiation, competence) and the resulting data used to evaluate the utility of the observational integrity measure for discriminating between EBPs.
Five psychometric properties of the TIMECS will be evaluated: content validity of the items (Phase I), inter-rater reliability of each item (Phase I, II, III), construct validity (convergent, discriminant) of the scales (Phase I, II), sensitivity of the items and scales (Phase II, III), and generalizability study (Phase II, III). Content validity will be determined by literature review to identify instructional practices to be transformed into practice elements. A consultant team will review as well as additional experts in the early childhood field to arrive at expert consensus about the items. To asses initial score reliability (inter-rater, test-retest) and validity (construct, accuracy), the team will code real-time observations from child participants with the TIMECS, and teacher ratings of the child participants (TIMECS-TR). To evaluate the generalizability and sensitivity of the observational and teacher report, the team will again code real-time observations from child participants with the TIMECS, and ask the teachers of the child participants to complete the TIMECS-TR. To evaluate the utility of the TIMECS, the research team will code recordings from four different early childhood interventions (BiC, First Step to Success, Teaching Pyramid, and Business as Usual) to evaluate the sensitivity of the observational measure at the item level.
Control Condition: In the control condition (business as usual), students receive standard classroom practices in place at the school.
Key Measures: Observational measures include the BEST in CLASS Adherence and Competence Scales (BiCACS; Sutherland et al., 2013) designed to assess integrity of the BiC program and is collected in the BiC Efficacy Trial, the Therapy Process Observational Coding System for Child Psychotherapy–Alliance scale (TPOCSA),and the Child Involvement Rating Scale (CIRS). Teacher report of the quality of the teacher-student relationship is measures with the Student Teacher Relationship Scale (STRS).
Data Analytic Strategy: In Phase I, researchers will use the Delphi technique to establish expert consensus. Mixed model procedures to account for the nested design will be used to determine inter-rater reliability, and person correlations will be used to assess construct, convergent, and discriminant validity. For Phase II, researchers will conduct a generalizability study to determine how many observations are needed to be collected to produce reliable estimates across different sources of variation in each TIMECS and TIMECS-TR scale. Researchers will assess sensitivity by evaluating differences in integrity between BiC and Business as usual by showing significant mean differences between treatments on the TIMECS and TIMESCS-TR. In Phase III, researchers will conduct a generalizability study to identify sources of variation in each TIMECS scale (3 EBPs, BAU). Sensitivity will be assessed by evaluating differences in integrity across 4 treatments (3 EBPs, BAU) by showing significant mean differences between treatments on TIMECS items unique to each EBP.
Journal article, monograph, or newsletter
McLeod, B.D., Sutherland, K.S., Martinez, R.G., Conroy, M.A., Snyder, P.A., and Southam-Gerow, M.A. (2017). Identifying Common Practice Elements to Improve Social, Emotional, and Behavioral Outcomes of Young Children in Early Childhood Classrooms. Prevention Science, 18(2): 204–213.