Co-Principal Investigators: Sam, Ann; Reska, Stephanie

Purpose: Social-emotional learning is a primary focus area in many publicly funded preschool programs, with increased emphasis placed on this school readiness domain because of the social isolation from peers that many young children have experienced during the COVID-19 pandemic. The purpose of this project is to establish the psychometric feature and instrument usability of a practitioner-administered observer impression scale assessment of preschool children's peer-related social competence. The Teacher Impression Scale (TIS) is an observer rating scale that teachers will be able to use in their classrooms to document the strengths that many children may possess in peer social competence, locate children in their classes who may be in need of more support for being socially engaged with their peers (i.e., children need Tier 2 supports in a Multi-Tiered Systems of Support-MTSS model), and monitor progress when additional support is provided to those children.

Project Activities: The research team will revise an observer impression scale to assess peer social competence of preschool children, establish the psychometric evidence of reliability and validity, and determine its usability for teachers in early learning programs in the U.S. This project will employ classical measurement theory and an iterative process for instrument development. Phase 1 of the project will focus on content validity, cultural and ethnic appropriateness of items, and instrumentation that would allow for online data collection. Phase 2 of the project will involve large-scale data collection that will allow instrument modification, confirmatory factor analysis, and evaluation of usability. In addition, two exploratory aims will be to examine the psychometric features of the instrument when used with diverse groups of children from different racial, ethnic, and cultural contexts, and the presence of rater bias. Phase 3 of the project will examine criterion-related validity, utility as a progress monitoring tool, interactions of child and rater demographic characteristics, and scalability. At the end of this project, a fully developed, psychometrically verified, and practical assessment of preschool children's peer social competence, suitable for scaling up for program use, will be available to early childhood programs and practitioners.

Products: By the end of the project, a fully developed TIS with strong evidence of reliability, validity, and usability will be available for teachers in early learning classes along with a brief 30-minute introduction to procedures for data collection and use of the data once collected. Current plans are to post the TIS for teacher and program use at no cost.

Structured Abstract

Setting: The study will take place in Head Start and state PreK programs in North Carolina and southern California. There will be a mix of urban, suburban, small town, and rural locations.

Sample: Across studies, 210 teachers in early learning classrooms and 740 preschool children will participate in the study. The teachers and children will be racially, ethnically, linguistically, and culturally diverse. Inclusion criteria are that children be within the 3–5 age range in the early learning programs, and do not have, and are not at-risk for, an identified developmental disability.

Assessment: The assessment is a practitioner-administered observer impression scale assessment of preschool children's peer-related social competence. The TIS is a 16-item observer rating scale, with each item having a 1 (item not observed) to 5 (item frequently observed) rating. The ratings for this scale are based on three 5-minute observations of preschool children engaging in social interaction with their peers. Teachers or other practitioners observe a child for a five minute sample during a free-play time in their classroom and complete the scale immediately after the observation. The information may be used for general assessment for all children, screening for children who may need support in establishing positive peer social competence, and progress monitoring.

Research Design and Methods: In the initial phase of the study, a Content Validity Index (CVI) analysis by four constituent groups (developmental scientists, leaders with expertise in racial, ethnic, and cultural responsiveness, practitioners, and family members) followed by virtual focus groups will provide initial information for instrument revisions. In the second phase, both interrater exact agreement percentage, test-retest reliability between raters, intra-class correlation coefficients examining number of observations, measurement of internal consistency, and confirmatory factor analyses will allow for a second revision of the instrument. The revised instrument will then be employed with a larger sample, with the same analyses repeated and also with examinations of test-retest reliability and concurrent validity. For the second set of data, factor structure and internal consistency will be examined for children from specific racial, ethnic, and cultural groups, and rater bias will be explored descriptively. The usability of the TIS for identifying or confirming the identification of children in need of support for peer social competence (which also provides a demonstration of criterion-related validity) and as a progress monitoring tool will be examined using latent growth curve analyses. Using an ingredients model, costs will be calculated for a program using the TIS for three different functions.

Control Condition: Due to the nature of this research there is no control condition.

Key Measures: Measure development will focus on the TIS item development work and validation work. To examine the concurrent validity of the TIS, teachers will complete the Social Skills Improvement System (SSIS).

Data Analytic Strategy: The research team will use a variety of analyses to examine the factor structure, internal consistency, validity, reliability, and usability of the Teacher Impression Scale. Methods include multilevel confirmatory factor analysis, analysis of variance, and multi-level latent growth curve models.

Cost Analysis: Using an ingredients model, researchers will calculate the costs for a program using the TIS for three different functions.