Grant Closed

Testing the Efficacy of Double Check: A Cultural Proficiency Professional Development Model in Middle Schools

NCER

Program: Education Research Grants

Program topic(s): Social, Emotional, and Behavioral Context for Teaching and Learning

Award amount: $3,496,525

Principal investigator: Catherine Bradshaw

Awardee:

University of Virginia

Year: 2015

Project type:

Efficacy

Award number: R305A150221

Purpose

In this study, researchers tested whether Double Check, a teacher professional development program designed to promote cultural proficiency and student engagement practices, can improve student academic and behavioral outcomes in middle school. Double Check was developed with IES funding (Double Check: A Cultural Proficiency and Student Engagement Model) as a potential way to reduce the disproportionate representation of minority students in disciplinary actions and special education referrals. Double Check provides school-wide training and individualized coaching using the Classroom Check-Up to teachers within a school-wide positive behavioral interventions and supports (SW-PBIS) framework. The Double Check model helps to integrate culturally responsive practices into middle schools as a way to strengthen student engagement in school and help teachers see student behavior through a different lens. These changes could help reduce unwarranted disciplinary actions and special education referrals.

Project Activities

The research team randomly assigned four consecutive cohorts of middle schools with SW-PBIS already in place to receive Double Check or to continue with their typical professional development opportunities. Intervention and data collection occurred with sixth and seventh grade students and their teachers. The researchers assessed the impact of Double Check on teacher attitudes and practices, student perceptions of school climate and equity, and student behavior and academic performance immediately following a 1-year intervention period and in the following school year.

Structured Abstract

Setting

This study was conducted in urban and suburban school districts in Maryland.

Sample

Grade 6 and 7 students (N= 6,276) and their teachers (N=352) participated across 41 middle schools.

Intervention

Double Check includes professional development for all school staff and coaching for classroom teachers within a school-wide positive behavioral interventions and supports (SW-PBIS) framework. The training is delivered in five sessions over the school year, with two booster training sessions in the following year to promote sustainability. Training content addresses five Double Check "CARES" core strategies: making Cultural connections to curriculum, developing Authentic relationships with students, Reflective thinking about group membership and function behind students' behavior, Effective communication with parents and students, and Sensitivity to students' cultures. Supplementary materials include "Double Checkers" — two-page summaries of a specific skill or strategy (such as Function-Based Thinking, Code Switching, and Perspective Taking) and the "Double Check Tip of the Week" provided to all school staff via email or staff newsletter. Individualized coaching is provided using the Classroom Check-Up coaching model with goal setting and performance feedback that reflects the five core modules.

Research design and methods

The research team randomly assigned middle schools that were already trained in SW-PBIS to receive training and coaching in the Double Check model or to serve as controls. They enrolled schools in the project across a series of four cohorts. Intervention and data collection focused on students and teachers in sixth and seventh grade classrooms. The team provided five sessions of active training and coaching over 1 year for teachers in these classrooms with 2 booster professional development sessions in the subsequent year to promote sustainability. The study team assessed student and teacher outcomes at the end of the intervention year and in the following school year to determine if impacts were sustained over time.

Control condition

Schools randomly assigned to the control condition already had SW-PBIS in place and provided typical professional development opportunities.

Key measures

The researchers assessed student outcomes through school records (standardized test scores, office discipline referrals, and attendance); observations using the ASSIST (Assessing School Settings: Interactions of Students and Teachers) and the CLASS (Classroom Assessment Scoring System); and student surveys focused on engagement, climate, equity, and teacher support. The team measured teacher outcomes using self-report surveys of culturally responsive efficacy, behavior management practices, and school climate and with observations using the ASSIST and the CLASS. In addition, they measured implementation using researcher-developed tools (logs, feedback forms, and fidelity checklists).

Data analytic strategy

The researchers used multilevel models to determine the impact of the school-level intervention on student and teacher outcomes. The team tested the potential moderating effects of teacher characteristics (e.g., race/ethnicity, gender) by expanding the main effects models to include cross-level interactions between teacher characteristics and study condition. They tested hypothesized mediators (culturally responsive teaching practices and proactive behavior management skills) using random coefficient growth modeling and its reformulation as a latent growth model. The team also examined variations in outcomes based on fidelity of implementation.

Key outcomes

The main findings of this study are as follows:

People and institutions involved

IES program contact(s)

Emily Doolittle

Team Lead for Social Behavioral Research

NCER

Products and publications

ERIC Citations: Find available citations in ERIC for this award here.

Publicly Available Data:

Bradshaw, C., Debnam, K., Pas, E., Rosenberg, M., Bottiani, J., Herman, K., Reinke, W., Kaihoi, C. (2023). 'Testing the Efficacy of Double Check: A Cultural Proficiency Professional Development Model in Middle Schools," University of Virginia Dataverse, V1 https://dataverse.lib.virginia.edu/dataset.xhtml?persistentId=doi:10.18130/V3/UCJOGL

Additional online resources and information:

This site is where you can find the materials that comprise the Double Check intervention that was tested in this project: https://doublecheckcoaching.org/.

Select Publications:

Bottiani, J. H., Bradshaw, C. P., & Gregory, A. (2018). Nudging the gap: Introduction to the special issue "Closing in on Discipline Disproportionality." School Psychology Review, 47 (2), 109-117. doi.org/10.17105/SPR-2018-0023.V47-2

Bottiani, J. H., Kush, J. M., McDaniel, H. L., Pas, E. T., & Bradshaw, C. P. (2023). Are we moving the needle on racial disproportionality? Measurement challenges in evaluating school discipline reform. American Educational Research Journal, 60(2), 293-329. doi.org/10.3102/00028312221140026

Bottiani, J.H., Bradshaw, C.P., & Mendelson, T.M. (2016). Inequality in Black and White high school students' perceptions of school support: An examination of race in context. Journal of Youth and Adolescence, 45(6), 1176-1191. doi.org/10.1007/s10964-015-0411-0

Bottiani, J.H., Bradshaw, C.P., & Mendelson, T.M. (2017). A multilevel examination of racial disparities in high school discipline: Black and White adolescents' perceived equity, school belonging, and adjustment problems. Journal of Educational Psychology, 109(4), 532-545.doi: dx.doi.org/10.1037/edu0000155.

Bottiani, J.H., Duran, C.A.K., Pas, E.T., & Bradshaw, C.P. (2019). Teacher stress and burnout in urban middle schools: Associations with job demands, resources, and effective classroom practices. Journal of School Psychology, 77, 36-51. doi.org/10.1016/j.jsp.2019.10.002.

Bottiani, J.H., Larson, K. Debnam, K., Bischoff, C., and Bradshaw, C.P. (2017). Promoting educators' use of culturally responsive practices: A systematic review of in-service interventions. Journal of Teacher Education, 1-19. doi.org/10.1177/0022487117722553

Bottiani, J.H., Lindstrom Johnson, S., McDaniel, H.L., & Bradshaw, C.P. (2019). Triangulating school climate: Areas of convergence and divergence across multiple levels and perspectives. American Journal of Community Psychology, 65, 423-436. doi.org/10.1002/ajcp.12410

Bottiani, J.H., McDaniel, Henderson, L., Castillo, J., & Bradshaw, C.P. (2020). Buffering effects of racial discrimination on school engagement: The role of culturally responsive teachers and caring school police. Journal of School Health, 1019-1029. doi.org/10.1111/josh.12967

Bradshaw, C. P, Pas, E. T., Bottiani, J., Debnam, K. J., Reinke, W., Herman, K., and Rosenberg, M. (2018). Promoting cultural responsivity and student engagement through Double Check coaching of classroom teachers: An efficacy study. School Psychology Review, 47, 118-134.

Bradshaw, C.P., Braun, S., McDaniel, H., Kaihoi, C., Pas, E., Bottiani, J.H., Cash, A., Debnam, K.J. (2023). Examining the psychometrics and characteristics of the Assessing School Settings: Interactions of Students and Teachers (ASSIST). Assessment for Effective Intervention, 1-12. doi.org/10.1177/1534508423118708

Camacho, D. Moore, S., Pas, E.T., & Bradshaw, C.P. (2022). Interactional quality in middle schools: Latent profiles and their associations with teacher, classroom, and school compositional factors. Journal of School Psychology, 93, 79-97. doi.org/10.1016/j.jsp.2022.06.001

Debnam , K. J., Pas, E. T., Bottiani, J., Cash, A. H., & Bradshaw, C. P. (2015). An examination of the association between self-reported and observed culturally responsive teaching practices. Psychology in the Schools, 52(6), 533-548. doi.org/10.1002/pits.21845

Debnam, K. J., Bottiani, J. H., & Bradshaw, C. P. (2017). Promoting culturally responsive practice to reduce disparities in school discipline among African American students. In N. Finnigan-Carr (Ed.), Linking Health and Education for African American Students' Success (pp. 97-114). New York, NY: Routledge Press.

Debnam, K. J., Milam, A., Bottiani, J., & Bradshaw, C. P. (2021). Teacher-student incongruence in perceptions of school equity: Associations with student perceived connectedness in middle and high school. Journal of School Health, 91(9), 706-713. doi.org/10.1111/josh.13062

Franco, M. P., Bottiani, J. H., & Bradshaw, C. P. (2024). Teachers' structuring of culturally responsive social relations and secondary students' experience of warm demand. Contemporary Educational Psychology, 76, 102241. doi.org/10.1016/j.cedpsych.2023.102241

Franco, M. P., Bottiani, J. H., & Bradshaw, C. P. (in press). Assessing teachers' culturally responsive classroom practice in PK-12 schools: A systematic review of teacher-, student-, and observer-report measures. Review of Educational Research. DOI: 10.3102/00346543231208720

Gaias, L. M., Lindstrom Johnson, S., Bottiani, J. H., Debnam, K. J., & Bradshaw, C. P. (2019). Examining teachers' classroom management profiles: Incorporating a focus on culturally responsive practice. Journal of School Psychology, 76, 124-139. doi.org/10.1016/j.jsp.2019.07.017

Henderson Smith, L., Bottiani, J.H., Kush, J., & Bradshaw, C.P. (2023). The discipline gap in context: The role of school racial and ethnic diversity and within school positionality on out-of-school suspensions. Journal of School Psychology, 98, 61-77. doi.org/10.1016/j.jsp.2023.02.006

Johnson, S. R., Pas, E. T., Bradshaw, C. P., & Ialongo, N. S. (2018). Promoting teacher implementation of classroom-based prevention programming through coaching: The mediating role of the coach-teacher relationship. Administration and Policy in Mental Health and Mental Health Services Research, 45, 404-416. doi.org/10.1007/s10488-017-0832-z

Johnson, S.R., Pas, E.T., and Bradshaw, C.P. (2016). Understanding and measuring coach-teacher alliance: A glimpse inside the 'black box'. Prevention Science, 17(4), 439-449. doi.org/10.1007/s11121-016-0633-8

Kaihoi, C.A., Bottiani, J. H., & Bradshaw, C. P. (2022). Teachers supporting teachers: A social network perspective on collegial stress support and emotional wellbeing among elementary and middle school educators. School Mental Health, 14, 1070-1085. doi.org/10.1007/s12310-022-09529-y

Kaihoi, C.A., Braun, S., Bottiani, J. H., & Bradshaw, C. P. (2023). Contextual factors contributing to variability within middle school teachers' observed classroom management and student behavior and engagement. Psychology in the Schools, 60(10), 4117-4142. doi.org/10.1002/pits.22995

Larson K.E., Bottiani, J.H., Pas, E.T., Bradshaw, C.P. (2019). A multilevel analysis of racial discipline disproportionality: A focus on student perceptions of academic engagement and disciplinary environment. Journal of School Psychology, 77, 152-167. doi.org/10.1016/j.jsp.2019.09.003

Larson, K. E., Pas, E. T., Bottiani, J. H., Kush, J. M., & Bradshaw, C. P. (2021). A multidimensional and multilevel examination of student engagement and secondary school teachers' use of classroom management practices. Journal of Positive Behavior Interventions, 23 (3), 149-162. doi.org/10.1177/1098300720929352

Larson, K., Pas. E. T., Bradshaw, C. P., Rosenberg. M. S., and Day-Vines, N. L. (2018). Examining how proactive behavior management and culturally responsive teaching relate to student behavior: Implications for measurement and practice. School Psychology Review, 47, 153-166. doi.org/10.17105/SPR-2017-0070.V47-2

McDaniel, H. L., Braun, S., Bottiani, J. H., De Lucia, D., Tolan, P., & Bradshaw, C. (2022). Examining developmental differences in teachers' observed classroom management strategies across elementary, middle, and high school. School Psychology Review, 1-21. doi.org/10.1080/2372966X.2022.2100275

Pas, E T., Borden, L., Herman, K., & Bradshaw, C. P. (2021). Leveraging Motivational Interviewing to coach teachers in the implementation of preventive evidence-based practices: A sequential analysis of the Motivational Interviewing process. Prevention Science, 22(6), 786-798. doi.org/10.1007/s11121-021-01238-3

Pas, E. T., Borden, L., Debnam, K. J., De Lucia, D., & Bradshaw, C. P. (2022). Exploring profiles of coaches' fidelity to Double Check's Motivational Interviewing-embedded coaching: Outcomes associated with fidelity. Journal of School Psychology, 92, 285-298. doi.org/10.1016/j.jsp.2022.04.003

Pas, E. T., Johnson, S. R., Debnam, K. J., Hulleman, C., & Bradshaw, C. P. (2019). Examining the relative utility of PBIS implementation fidelity scores in relation to student outcomes. Remedial and Special Education, 40(1), 6-15. doi.org/10.1177/0741932518805192

Pas, E. T., Kaihoi, C. A., Debnam, K. J., & Bradshaw, C. P. (2022). Is it more effective or efficient to coach teachers in pairs or individually? A comparison of teacher and student outcomes and coaching costs. Journal of School Psychology, 92, 346-359. doi.org/10.1016/j.jsp.2022.03.004

Pas, E. T., Larson, K. E., Reinke, W. M., Herman, K. C., and Bradshaw, C. P. (2016). Implementation and acceptability of an adapted Classroom Check-Up coaching model to promote culturally-responsive classroom management. The Education and Treatment of Children, 39, 467-492. doi.org/10.1353/etc.2016.0021

Related projects

Supplemental information

Co-Principal Investigator: Debnam, Katrina

In classrooms where teachers received high-fidelity coaching in the Double Check model, there were fewer instances of student non-cooperation. Although the impact was not statistically significant, teachers who received the coaching were less reactive in their approach to behavior management and provided students with more opportunities to respond (Pas, Borden, Debnam, de Lucia, and Bradshaw, 2022).
When coaches used language that was consistent with motivational interviewing, teachers were more likely to express desire for, interest in, and readiness for change (Pas, Borden, Herman, and Bradshaw, 2021).

Cost/Cost Effectiveness Analysis: The analytic team calculated the total cost of the intervention based on the expenditures/outlays for each of its components (e.g., intervention protocols/materials, personnel costs). The total average cost of intervention per teacher was $551.57 in 2023 USD among all intervention condition teachers (i.e., the full sample of teachers in the intervention schools including those who did not implement any elements of the program), or about $621.51 in 2023 USD among intervention teachers who participated in at least some of the intervention activities.

Questions about this project?

To answer additional questions about this project or provide feedback, please contact the program officer.