Co-Principal Investigator: Dana McCoy (Harvard University)

Purpose: The goal of this project was to identify variables that create differences in impacts of early childhood professional development interventions. Achieving this aim will support decision making around scaling and refining professional development programs in order to optimize classroom quality and child learning outcomes.

Project Activities: This was a retrospective study of data already collected through the National Center for Research on Early Childhood Education Professional Development Study (NCRECE PDS). The primary activities of the grant were analyzing the data for a 4-year span (2008–2011), interpreting the results, and disseminating the findings to academic and policy audiences.

Key Outcomes: The main findings of this project are as follows:

• The effect of the NCRECE PDS interventions did not vary across centers for most domains of teacher-child interactions and child outcomes; the only exception was that the programs' impact on classroom organization did, in fact, appear to vary significantly across centers (Sabol et al., 2022).
• The team also adapted and validated a new tool, the internet-based School Neighborhood Assessment Protocol (iSNAP) that uses a systematic social observation protocol to assess the physical condition of the neighborhood using Google Street View (see McCoy et al. 2022 for a description of neighborhood measures).
• Characteristics inside school walls—including the demographic characteristics of students within the school, structural quality, and process quality— were not associated with the treatment impact on student outcomes. The only exception was that the interventions used in the NCRECE PDS positively impacted child outcomes in preschools where educators collectively reported high workplace stress, but negatively impacted child outcomes in preschools where educators collectively reported low workplace stress (Hanno et al, 2021).
• Some characteristics of contexts outside school walls were found to predict treatment impacts. The interventions used in the NCRECE PDS had a significantly more positive effect on children's executive functioning in preschools that were in neighborhoods with lower levels of institutional resources (e.g., less greenspace) and lower structural/social processes (e.g., worse physical condition of the local environment) compared to preschools with higher levels of the same neighborhood processes. (McCoy, et al., 2023).
• The research team concluded that neighborhood-centered information can offer additional information on the effectiveness of teacher professional development, helping to inform how and under what conditions to bring such interventions to scale. However, the team also concluded that characterizing and predicting treatment impact variation was not without its challenges and requires care to select the best methods for analyzing impact variation and to prospectively collect data on the characteristics of sites that may help to explain this variability (these lessons are summarized in Sabol et al., 2022).

Structured Abstract

Setting: This study used existing data from the NCRECE PDS, which took place in schools in nine socio-demographically diverse cities in the United States.

Sample: The full analytic study sample included 1,134 children, 263 teachers, and 163 centers in the course phase of the intervention, and 1,407 children, 326 teachers, and 198 centers in the coaching phase of the intervention. The sample, predominantly low-income and racially/ethnically diverse, is designed to represent the population of children in community and publicly-funded preschool programs (e.g., Head Start) in the United States.

Intervention: The NCRECE PDS evaluated the independent and additive impacts of two professional development components designed to enhance teacher-child interactions: (1) a 14-week course and (2) individualized coaching. The course was designed to increase in-service teachers' knowledge about the importance of teacher-child interactions to help promote children's learning and skill acquisition. The coaching occurred through a web-based system onto which teachers uploaded videos of their teaching. After watching the videos, coaches helped teachers to identify effective teaching behavior, make connections between teacher and child behaviors, and problem solve to determine alternative strategies when necessary.

Research Design and Methods: In the NCRECE PDS, the interventions were randomly assigned to teachers via a 2x2 sequential design. In Phase I, teachers were randomly assigned either to receive the 14-week professional development course or to be in the control group. In Phase II, teachers were again randomized, this time either to receive coaching or not receive coaching. This process generated four experimental groups: (1) no course/no coaching (i.e., control); (2) course/no coaching; 3) no course/coaching; and (4) course/coaching. Findings from an intent-to-treat analysis indicated that the course and coaching, both separately and together, led to increases in teachers' use of effective social and instructional interactions. Regarding child outcomes, the course component and the coaching component separately were found to improve child outcomes for expressive language and self-regulation skills. In this study, the researchers expanded the evaluation of these main effects to explore variation in impacts and potential predictors of this variation via multilevel modeling.

Control Condition: Teachers in the control condition were followed across time but did not receive any professional development services.

Key Measures: Classroom quality outcomes were measured by the Classroom Assessment Scoring System™. Child language, emergent literacy skills, and self-regulation were measured using the PPVT, WJ-III Picture Vocabulary, the Phonological Awareness and Print Knowledge subtests of the Test of Preschool Early Literacy, Pencil Tap, and the Backward Digit Span. Data for contextual predictors of impact variation within and outside of school were also used.

Data Analytic Strategy: Researchers used fixed intercept random coefficient (FIRC) approaches to evaluate impact variation in the NCRECE PD study and to explore whether variation in treatment effects was predicted by characteristics within school walls and characteristics outside of school walls.  To render the analyses more manageable in terms of the number of predictors in a given model, data reduction techniques, such as factor analysis and cluster analysis were used to combine or eliminate predictors.

Products and Publications

Additional Online Resources: Website for iSNAP: https://sites.northwestern.edu/isnap/

ERIC Citations: Find available citations in ERIC for this award here.

Select Publications:

Hanno, E. C. (2022). Immediate Changes, Trade-Offs, and Fade-Out in High-Quality Teacher Practices during Coaching. Educational Researcher, 51(3), 173–185.

Hanno, E., McCoy, D. C., Sabol, T. J., & Gonzalez, K. E. (2021). Early educators' collective workplace stress as a predictor of professional development's impacts on children's development. Child Development, 92(3), 833–843. https://doi.org/10.1111/cdev.13566 

Hanno, E. C. & Gonzalez, K. E. (2020). The effects of teacher professional development on student attendance in preschool. Journal of Research on Educational Effectiveness, 13(1), 3–28. doi:10.1080/19345747.2019.1634170

McCoy, D. C., Sabol, T. J., Hanno, E. C., & Odgers, C. L. (2022). Assessing School Communities Using Google Street View: A Virtual Systematic Social Observation Approach. AERA Open, 8(1).

McCoy, D. C., Sabol, T. J., Wei, W., Busby, A., & Hanno, E. C. (2022). Pushing the boundaries of education research: A multidimensional approach to characterizing preschool neighborhoods and their relations with child outcomes. Journal of Educational Psychology, 115(1), 143–159. https://doi.org/10.1037/edu0000728

Sabol, T. J., McCoy, D., Gonzalez, K., Miratrix, L., Hedges, L., Spybrook, J. K., & Weiland, C. (2022). Exploring treatment impact heterogeneity across sites: Challenges and opportunities for early childhood researchers. Early Childhood Research Quarterly, 58, 14–26. https://doi.org/10.1016/j.ecresq.2021.07.005

Wei, W.S., McCoy, D.C., Busby, A.K., Hanno, E.C., & Sabol, T.J. (2021). Beyond neighborhood socioeconomic status: Exploring the role of neighborhood resources for preschool classroom quality and early childhood development. American Journal of Community Psychology, 67:1–16. https://doi.org/10.1002/ajcp.12507