Co-Principal Investigators: Bradshaw, Catherine P.; Herman, Keith

Purpose: The purpose of this project is to extend, refine, and validate a measurement system to assess culturally- and racial equity-sustaining (CARES) practices in the classroom, leveraging the model that was previously developed with the same acronym to highlight its five relevant domains: Connections to curriculum, Authentic relationships, Reflective thinking, Effective communication, and Sensitivity to student culture. The CARES classroom assessment system (including three instruments initially developed in prior IES Exploration and related IES Development and Efficacy projects) is designed to support teachers in elementary and middle schools to improve their CARES instruction and discipline. The team will iteratively refine and validate the multi-rater CARES classroom assessment system to be a streamlined, psychometrically sound, and useful to teachers as a source of professional development feedback. This project addresses important and timely issues related to culturally-responsive practices and equity in education; it also addresses considerable gaps related to the lack of valid, reliable, and efficient methods for assessing CARES, which in turn can help to improve the social and behavioral context for academic learning and reduce a range of disparate educational outcomes (e.g., discipline, academic) for Black, Indigenous, and other students of color.

Project Activities: Researchers will address the following aims. In Aim 1, the research team will conceptually update, align, and validate the separate teacher-, student-, and observer-report instruments. In Aim 2, the integrated CARES multi-rater assessment tool will be psychometrically validated. In Aim 3, the acceptability, feasibility, and utility of the CARES multi-rater assessment tool will be optimized for informing teacher professional development.

Products: The primary product isa integrated, multi-rater CARES classroom assessment system, which includes teacher-report, student-report, and observer-report instruments. The system will also include teacher-level score reports and interpretation guidance.

Structured Abstract

Setting: The multi-disciplinary multi-method research team will draw upon researchers at the University of Virginia and the University of Missouri, working in close partnership with schools in two districts in Maryland (Anne Arundel County Public Schools and Howard County Public Schools) and one district in Missouri (Jefferson City Public Schools).

Sample: Over four years, the CARES classroom assessment system will be iteratively refined and piloted with approximately 372 staff and 3,720 students sampled from 12 schools in Virginia and Missouri to be representative of diverse racial and ethnic compositions.

Assessment: The CARES classroom assessment system will be a psychometrically valid and reliable, feasible and acceptable, and ultimately useful multi-informant tool that captures key domains and dimensions of culturally- and racial equity-sustaining classroom instruction and discipline practices. The final product will include three validated instruments (teacher-, student-, and observer-report), and accompanying resources and materials to support the interpretation and use of scores that result from the assessment. These accompanying materials will include a template for teacher-level score reports and interpretation guidance.

Research Design and Methods: Across the three project aims, researchers will complete a series of five studies using an iterative mixed methods design with rigorous qualitative and quantitative analyses to extend, refine, and streamline a multi-rater CARES classroom assessment system. In Study 1, the team will conduct cognitive interviews and focus groups with students, teachers and parents to examine content validity of newly developed CARES items focused on racial equity. In Study 2, researchers will complete a pilot of the multi-rater aligned and updated instruments to establish strong psychometric properties for each instrument, nomological validity, and convergent and divergent validity within the instrument framework and theory of change. In Study 3, the team will evaluate convergent validity when triangulating across raters in a second wave of pilot data collection. In Study 4, researchers will examine differential item functioning in various classroom racial and ethnic compositions of teacher and student (mis)match using a third wave of pilot data. In Study 5, the team will focus on feasibility and acceptability testing with end users (teachers).

Control Condition: This project is not experimental; as such, there is no control condition.

Key Measures: The key measures are the ones proposed for development and validation, the CARES teacher-, student- and observer-report instruments. Other measures include the student reported social-emotional, engagement, and academic outcomes (GPA, standardized test scores), and costs.

Data Analytic Strategy: Researchers will conduct mixed-method qualitative and quantitative data analyses for studies 1 and 5 to inform key revisions to the CARES classroom assessment system to improve its reliability, validity, and teacher receptivity to its use as a source of professional development feedback. In studies 2–4, the team will use a combination of factor analytic, item response theory (IRT), and multi-trait multi-method matrix approaches, including multilevel confirmatory factor analyses and moderated non-linear factor analyses.

Cost Analysis: The team will analyze costs of the CARES classroom assessment system training and administration using an "ingredients" approach and the CostOut program to track and estimate total costs for assessment, including observer training, administration, and score reporting.

Related IES Projects: Identifying Discrete and Malleable Indicators of Culturally Responsive Instruction and Discipline (R305A180111); Double Check: A Cultural Proficiency and Student Engagement Model (R324A110107); Testing the Efficacy of Double Check: A Cultural Proficiency Professional Development Model in Middle Schools (R305A150221).