Co-Principal Investigators: Smith, Toni; Griffin, Melinda; Song, Mengli; Danks, Amanda

Purpose: The Simulated Instruction in Mathematics Professional Development (SIM PD) program is an innovative professional development (PD) program that offers (1) a focus on research-based math instructional strategies and (2) the unique opportunity to practice implementing these strategies in a mixed-reality environment with support of a coach and teacher colleagues. The program responds to the critical need in the field for ways to develop math teachers' ability to use the focal strategies, which help students (including those from underserved, underrepresented groups) to develop a deep understanding of math concepts, effectively communicate their thinking, and use reasoning to solve math problems. This initial efficacy study also will help address the paucity in research about effective math PD programs.

A prior 16-school randomized pilot study found large, significant, positive impacts of SIM PD on the quality of classroom math discussion (effect size = 1.08). It also generated evidence that students of teachers who participated in SIM PD provided more mathematical explanations and justifications that focused on math conceptual foundations and connections than their peers in the control condition. Building on the earlier pilot study, this proposed study will provide rigorous evidence on the efficacy of a promising PD program implemented on a larger scale. Because the program offers an immersive, authentic environment to practice instructional strategies, is delivered using video-conferencing technology, is geographically accessible, and is structured to provide consistent simulation experiences, it holds promise for providing high-quality PD to large numbers of teachers. The results of the project will provide much-needed evidence about the PD activities that improve instruction and student achievement.

Project Activities: The project will include 60 schools and 240 teachers serving 6,000 students in Grades 4 and 5. Researchers will use a cluster, multisite field trial that randomly assigns schools with equal probability to treatment—participation in SIM PD—or a delayed-treatment control condition. The study will collect classroom observations and end-of-year student performance on statewide math assessments to assess the impacts of SIM PD. The study also will examine implementation, service contrast, and costs, using information collected from PD observations, coaching logs, teacher surveys, district and school administrator interviews, and extant cost and pricing records. Last, the study will explore student perspectives through student focus groups.

Products: Researchers plan to disseminate the findings and implications for practice to a diverse audience through journal publications, research briefs, webinars, practitioner papers, conference presentations, and postings to social media.

Structured Abstract

Setting: The project will focus on districts that primarily serve underserved students. The first cohort of schools will be in a mid-sized urban district in New Mexico serving predominantly Latinx students and a large suburban district in Georgia serving predominantly African American students.

Sample: Teachers involved in the study will provide primary math instruction in Grades 4 and 5, serving students in general grade-level classes.

Intervention: SIM PD uses workshops, practice sessions, and group debriefing meetings to support teachers in using research-based questioning and discourse facilitation strategies. The activities are organized into three modules, each focused on a different instructional strategy. Within each module, teachers from the same school participate in workshops to learn about a focal strategy, and the research that supports it, and then prepare to practice it. During practice sessions in the mixed-reality classroom, teachers try the strategy with student avatars, pause to get feedback and reflect on progress with a coach and other teachers, identify next steps, and try again. Teachers practice in the mixed-reality classroom first with their school team and then individually. Last, teachers meet again to reflect on their experiences and discuss how they have and will continue to transfer their learning to their classrooms.

Research Design and Methods: Researchers will use a multisite, cluster randomized trial designed to meet What Works Clearinghouse standards without reservations. The trial will involve 60 schools and about 240 teachers across two cohorts. Schools within each cohort will be randomly assigned with equal probability to the treatment group or the delayed-treatment control group within each study district at the beginning (early fall) of the intervention year. Researchers also will conduct an implementation study using PD artifacts and observations, administrator interviews, and teacher surveys, and explore student experiences through student focus groups.

Control Condition: Participating teachers at schools in the control condition will continue with the business-as-usual PD offering available through their schools and districts. In the year after evaluation activities, these teachers will be offered participation in SIM PD.

Key Measures: To assess program impact on the quality of classroom math discussion, the evaluation team will collect video classroom observations at three time points for each teacher in the study: a baseline observation in the fall of the intervention year before SIM PD begins (fall 2023 for Cohort 1 and fall 2024 for Cohort 2), and two observations immediately after the PD ends in the spring (spring 2024 or 2025). Videos will be coded remotely by math instruction experts blind to each teacher's treatment condition, using scoring codes adapted from the Mathematics Scan observation rubric  that focus on features of mathematics group discussion.

To examine student math achievement for both cohorts, the evaluation team will collect fourth- and fifth-grade state math assessment scores from spring 2024 (Cohort 1) and spring 2025 (Cohort 2). Assessment scores across districts will be standardized using grade-level means and standard deviations of the test scores within each state.

Data Analytic Strategy: Impacts on the quality of classroom math discussion and on student achievement will be estimated using multilevel modeling. Researchers will explore potential moderation effects of key student and teacher characteristics, and examine whether the impact of SIM PD on student math achievement is mediated by the quality of classroom math discussion. To help interpret impact findings, researchers will assess implementation fidelity and explore how implementation fidelity is associated with teacher and student outcomes based on data from the treatment group.

Cost Analysis: Researchers will conduct cost analyses using a resource cost model that uses an "ingredients" approach to capture systematically all personnel and non-personnel costs for implementing an intervention. The cost-analysis results will be combined with the impact findings to conduct a cost-effectiveness analysis that captures the cost-per-standard-deviation impact of the PD. Cost-effectiveness will be calculated both per-teacher and per-student.