Project Activities
The research team will revise an observer impression scale to assess peer social competence of preschool children, establish the psychometric evidence of reliability and validity, and determine its usability for teachers in early learning programs in the U.S. This project will employ classical measurement theory and an iterative process for instrument development. Phase 1 of the project will focus on content validity, cultural and ethnic appropriateness of items, and instrumentation that would allow for online data collection. Phase 2 of the project will involve large-scale data collection that will allow instrument modification, confirmatory factor analysis, and evaluation of usability. In addition, two exploratory aims will be to examine the psychometric features of the instrument when used with diverse groups of children from different racial, ethnic, and cultural contexts, and the presence of rater bias. Phase 3 of the project will examine criterion-related validity, utility as a progress monitoring tool, interactions of child and rater demographic characteristics, and scalability. At the end of this project, a fully developed, psychometrically verified, and practical assessment of preschool children's peer social competence, suitable for scaling up for program use, will be available to early childhood programs and practitioners.
Structured Abstract
Setting
The study will take place in Head Start and state PreK programs in North Carolina and southern California. There will be a mix of urban, suburban, small town, and rural locations.
Sample
Across studies, 210 teachers in early learning classrooms and 740 preschool children will participate in the study. The teachers and children will be racially, ethnically, linguistically, and culturally diverse. Inclusion criteria are that children be within the 3-5 age range in the early learning programs, and do not have, and are not at-risk for, an identified developmental disability.
The assessment is a practitioner-administered observer impression scale assessment of preschool children's peer-related social competence. The TIS is a 16-item observer rating scale, with each item having a 1 (item not observed) to 5 (item frequently observed) rating. The ratings for this scale are based on three 5-minute observations of preschool children engaging in social interaction with their peers. Teachers or other practitioners observe a child for a five minute sample during a free-play time in their classroom and complete the scale immediately after the observation. The information may be used for general assessment for all children, screening for children who may need support in establishing positive peer social competence, and progress monitoring.
Research design and methods
In the initial phase of the study, a Content Validity Index (CVI) analysis by four constituent groups (developmental scientists, leaders with expertise in racial, ethnic, and cultural responsiveness, practitioners, and family members) followed by virtual focus groups will provide initial information for instrument revisions. In the second phase, both interrater exact agreement percentage, test-retest reliability between raters, intra-class correlation coefficients examining number of observations, measurement of internal consistency, and confirmatory factor analyses will allow for a second revision of the instrument. The revised instrument will then be employed with a larger sample, with the same analyses repeated and also with examinations of test-retest reliability and concurrent validity. For the second set of data, factor structure and internal consistency will be examined for children from specific racial, ethnic, and cultural groups, and rater bias will be explored descriptively. The usability of the TIS for identifying or confirming the identification of children in need of support for peer social competence (which also provides a demonstration of criterion-related validity) and as a progress monitoring tool will be examined using latent growth curve analyses. Using an ingredients model, costs will be calculated for a program using the TIS for three different functions.
Control condition
Due to the nature of this research, there is no control condition.
Key measures
Measure development will focus on the TIS item development work and validation work. To examine the concurrent validity of the TIS, teachers will complete the Social Skills Improvement System (SSIS).
Data analytic strategy
The research team will use a variety of analyses to examine the factor structure, internal consistency, validity, reliability, and usability of the Teacher Impression Scale. Methods include multilevel confirmatory factor analysis, analysis of variance, and multi-level latent growth curve models.
Cost analysis strategy
Using an ingredients model, researchers will calculate the costs for a program using the TIS for three different functions.
People and institutions involved
IES program contact(s)
Project contributors
Products and publications
By the end of the project, a fully developed TIS with strong evidence of reliability, validity, and usability will be available for teachers in early learning classes along with a brief 30-minute introduction to procedures for data collection and use of the data once collected. Current plans are to post the TIS for teacher and program use at no cost.
Publications:
ERIC Citations: Find available citations in ERIC for this award here.
Questions about this project?
To answer additional questions about this project or provide feedback, please contact the program officer.
