Project Activities
The project will be carried out in three phases and researchers will incorporate data collected from classrooms participating in a
previously funded IES Goal 3 grant, a randomized controlled trial evaluating the efficacy of the BEST in CLASS (BiC) early childhood program, to evaluate the initial psychometrics of the observational and teacher self-report integrity measures. In Phase I, the research team will develop and refine the treatment integrity measures, establishing score reliability and validity (content, construct). During Phase II, researchers will evaluate the generalizability and sensitivity of the integrity measures. The team will evaluate the utility of the observational measure during Phase III.
Structured Abstract
Setting
Data will be drawn from the one hundred and twenty early childhood classrooms across urban cities in Virginia which are participating in an existing BiC efficacy trial are included.
Sample
The Phase I sample will include all children enrolled in the VA site during year 4 of the BiC Efficacy Trial, including approximately 60 children (n=30 BiC; n=30 business as usual) and 20 teachers (n=10 BiC; n=10 business as usual). The Phase II sample will include approximately 120 children with problem behavior (n=60 BiC; n=60 business as usual) and 40 teachers (n=20 BiC; n=20 business as usual). The Phase III sample will include 72 children and 72 teachers.
Intervention
: BiC is a classroom-based program designed to ameliorate the problem behavior and increase the preacademic learning of young children with problem behavior. BiC is designed to increase the quantity and quality of evidence-based instructional practices implemented by teachers in early childhood classrooms. BiC comprises 10 instructional practices that are empirically linked to positive child behavioral and pre-academic outcomes. First Step to Success is an EBP for young children with problem behavior that includes school and home based components. The Teaching Pyramid intervention is a multi-tiered classroom-based EBP designed to promote social-emotional development and prevent challenging behavior of young children with problem behavior.
Research design and methods
Data from classrooms participating in a
previously funded IES Goal 3 grant, a randomized controlled trial evaluating the efficacy of the BiC early childhood program, will be used to evaluate the initial psychometrics of the observational and teacher self-report integrity measures. For Phase I and II, real-time classroom observations of children with problem behaviors will be coded using the observational treatment integrity measure; teachers will self-report on their use of instructional practices. Phase III will use videotaped recordings from children and teachers delivering 3 EBPs (Best in Class; First Step to Success; Teaching Pyramid) and a business and usual condition, which will be coded for integrity (adherence, differentiation, competence) and the resulting data used to evaluate the utility of the observational integrity measure for discriminating between EBPs.
Control condition
In the control condition (business as usual), students receive standard classroom practices in place at the school.
Key measures
Observational measures include the BEST in CLASS Adherence and Competence Scales (BiCACS; Sutherland et al., 2013) designed to assess integrity of the BiC program and is collected in the BiC Efficacy Trial, the Therapy Process Observational Coding System for Child Psychotherapy–Alliance scale (TPOCSA),and the Child Involvement Rating Scale (CIRS). Teacher report of the quality of the teacher-student relationship is measures with the Student Teacher Relationship Scale (STRS).
Data analytic strategy
In Phase I, researchers will use the Delphi technique to establish expert consensus. Mixed model procedures to account for the nested design will be used to determine inter-rater reliability, and person correlations will be used to assess construct, convergent, and discriminant validity. For Phase II, researchers will conduct a generalizability study to determine how many observations are needed to be collected to produce reliable estimates across different sources of variation in each TIMECS and TIMECS-TR scale. Researchers will assess sensitivity by evaluating differences in integrity between BiC and Business as usual by showing significant mean differences between treatments on the TIMECS and TIMESCS-TR. In Phase III, researchers will conduct a generalizability study to identify sources of variation in each TIMECS scale (3 EBPs, BAU). Sensitivity will be assessed by evaluating differences in integrity across 4 treatments (3 EBPs, BAU) by showing significant mean differences between treatments on TIMECS items unique to each EBP.
People and institutions involved
IES program contact(s)
Products and publications
Products: The products of this project will include fully developed and validated observational and teacher self-report treatment integrity measures designed to characterize the implementation of EBPs delivered by teachers in early childhood classrooms. Peer reviewed publications will also be produced.
ERIC Citations: Find available citations in ERIC for this award here.
Journal articles
Farmer, T. W., Sutherland, K. S., Talbott, E., Brooks, D. S., Norwalk, K., & Huneke, M. (2016). Special educators as intervention specialists: Dynamic systems and the complexity of intensifying intervention for students with emotional and behavioral disorders. Journal of Emotional and Behavioral Disorders,24(3), 173-186. Full text
McLeod, B. D., Sutherland, K. S., Broda, M., Granger, K. L., Cecilione, J., Cook, C. R., ... & Southam-Gerow, M. A. (2022). Examining the correspondence between teacher-and observer-report treatment integrity measures. School Mental Health,14(1), 20-34. Full text
McLeod, B.D., Sutherland, K.S., Martinez, R.G., Conroy, M.A., Snyder, P.A., & Southam-Gerow, M.A. (2017). Identifying common practice elements to improve social, emotional, and behavioral outcomes of young children in early childhood classrooms. Prevention Science, 18(2), 204-213. Full text
Sutherland, K.S. & McLeod, B.D. (2022). Advancing the science of integrity measurement in school mental health research. School Mental Health, 14(1), 1-6.
Sutherland, K.S., McLeod, B.D., Conroy, M.A., & McCormick, N. (2022). Developing treatment integrity measures for teacher-delivered interventions: Progress, recommendations and future directions. School Mental Health, 14(1), 7-19. Full text
Related projects
Supplemental information
Co-Principal Investigator: Kevin Sutherland
Assessment: The assessment system that will be created will evaluate three components of treatment integrity: (a) Treatment adherence—the extent to which a teacher delivers instructional practices "prescribed" by a EBP; (b) Treatment differentiation—the extent to whether (and “to where”) teachers deviate from a treatment protocol, and (c) Competence—the level of skill shown by the teachers in delivering the EBP. The system includes The Treatment Integrity Measure for Early Childhood Settings (TIMECS) which will assess the integrity of instructional practices delivered by teachers in early childhood classrooms. TIMECS will contain 3 scales: Adherence, Competence, and Differentiation. The Treatment Integrity Measure for Early Childhood Settings—Teacher Report (TIMECS-TR) will be developed to assess teacher-report of adherence and differentiation of instructional practices. TIMECS-TR will contain 2 scales: Adherence and Differentiation.
Five psychometric properties of the TIMECS will be evaluated: content validity of the items (Phase I), inter-rater reliability of each item (Phase I, II, III), construct validity (convergent, discriminant) of the scales (Phase I, II), sensitivity of the items and scales (Phase II, III), and generalizability study (Phase II, III). Content validity will be determined by literature review to identify instructional practices to be transformed into practice elements. A consultant team will review as well as additional experts in the early childhood field to arrive at expert consensus about the items. To asses initial score reliability (inter-rater, test-retest) and validity (construct, accuracy), the team will code real-time observations from child participants with the TIMECS, and teacher ratings of the child participants (TIMECS-TR). To evaluate the generalizability and sensitivity of the observational and teacher report, the team will again code real-time observations from child participants with the TIMECS, and ask the teachers of the child participants to complete the TIMECS-TR. To evaluate the utility of the TIMECS, the research team will code recordings from four different early childhood interventions (BiC, First Step to Success, Teaching Pyramid, and Business as Usual) to evaluate the sensitivity of the observational measure at the item level.
Questions about this project?
To answer additional questions about this project or provide feedback, please contact the program officer.
