Measuring Preschool Program Quality: Multiple Aspects of the Validity of Two Widely-Used Measures
Co-Principal Investigators: Kerry Hofer (Vanderbilt University); Everett Smith (University of Illinois at Chicago)
Purpose: The ability to measure the quality of preschool settings is of central importance in supporting a number of policy initiatives, such as state Quality Rating and Information systems, Head Start recompetition regulations, and the Race to the Top Early Learning Challenge. This project will study aspects of validity of two widely used measures of preschool classroom quality: the Early Childhood Environment System Rating Scale—Revised (ECERS-R) and the Classroom Assessment Scoring System (CLASS). Results will provide information for researchers and policymakers on the utility of these measures in characterizing the quality of preschool settings.
Project Activities: Researchers will use extant data to examine predictive validity specific to the cutoffs defined in policy systems and translate findings into metrics that can be used by researchers and policymakers. Structural validity will be studied to determine the extent to which each scale measures the desired dimensions. Content validity will be studied by determining whether the scores distinguish between lower, middle and higher quality settings. Finally, the utility of the measures for estimating children's school readiness will be studied.
Products: Products include findings on the utility of the ECERS-R and the CLASS as tools to characterize preschool classroom quality and peer-reviewed publications describing those findings.
Setting: The project will use twelve datasets, most drawn with national sampling frames or from multiple sites across the country. In each data set, only data for 3–5 year-old children who receive care or attend preschool in centers will be included. Depending on the dataset, these programs are located in public and private schools, churches, and non-profit or for-profit organizations.
Sample: Some datasets focus on at-risk subpopulations, including low-income children, Head Start participants, and state pre-kindergarten programs. Others are nationally representative but include a large number of low-income children.
Intervention: Widely used as a measure of global classroom quality, the ECERS-R was developed from a checklist of items amassed through the professional experience of the scale developers and is aimed at helping centers self-improve. The ECERS-R measure includes six subscales that are often combined to provide scores on three dimensions: (1) Language-Reasoning and Interaction, (2) Space and Furnishings, Activities, and Programs, and (3) Personal Care. In contrast, the CLASS was designed with conceptual domains in mind, three in the latest edition: Emotional Support, Classroom Organization, and Instructional Support. Each of these domains is built on theoretical and research evidence regarding how the domain supports children's outcomes. The CLASS developers have conducted confirmatory factor analyses to test for the three expected dimensions, concluding that there is modest support for the scale's structure:
Research Design and Methods: Research questions will be addressed by conducting secondary data analyses and meta-analyses using data from the following twelve datasets: four cohorts from the Head Start Family and Child Experiences Survey; Head Start Impact Study; National Center for Early Development and Learning Multistate Study of Prekindergarten; Preschool Curriculum Evaluation Research Initiative; Quality Interventions for Early Care and Education; Early Head Start Research and Evaluation Project; Welfare, Children and Families: A Three City Study; the Fragile Families and Child Wellbeing Study; and the Early Childhood Longitudinal Study, Birth-Cohort. Child outcomes in language and literacy, math, school readiness, approaches to learning, social competence, behavior problems, and health will be related to data from the two observation scales (CLASS and ECERS-R). Student, parent, and center covariates will also be included in the analyses.
Control Condition: Due to the nature of this study, there is no control condition.
Key Measures: The researchers will use secondary data sources to examine associations between the ECERS-R, CLASS, and child outcomes. The child outcome measures include assessments of children's cognitive and social behavioral skills, as well as indicators of physical health and well-being. For example, the Peabody Picture Vocabulary Test and Woodcock-Johnson, the Social Skills Rating System and Teacher Report Form are used.
Data Analytic Strategy: Structural validity will be studied using factor analysis and item response theory (IRT). Factor analysis will provide evidence of dimensionality and IRT will be used to test for order (whether higher scores reflect higher quality on the underlying dimension; fit (whether items go together to define an underlying dimension; and separation (the degree to which items can distinguish, low, moderate and high quality settings). Predictive validity will be evaluated using regression. Propensity scores and instrumental variables will be used to better isolate the true relationship between center quality and student achievement. A meta-analysis of studies that examine readiness gaps between lower and higher-income children will be conducted in order to determine if preschool classroom quality is associated with those gaps.