The New Mexico Public Education Department developed the Kindergarten Observation Tool (KOT) as a multidimensional observational measure of students' knowledge and skills at kindergarten entry. The primary purpose of the KOT is to inform instruction, so that kindergarten teachers can use the information about their students' knowledge and skills from the KOT to inform their curricular and pedagogical decisions. Stakeholders also are interested in using data from the KOT for other purposes, such as assessing student readiness for school statewide and identifying disparities in students' readiness for school across the state. This study examined the construct validity of the KOT to determine whether data from a field test supported using the KOT to measure six school readiness domains and, if not, what domain structure the data best supported. The study was conducted in response to the New Mexico Prekindergarten Research Alliance's and New Mexico Public Education Department's interest in evidence for the KOT's validity and reliability. The study team conducted exploratory and confirmatory factor analyses to identify the latent constructs that the KOT measured. The analyses identified the sets of items that were most related to one another and that could be used to develop domain scores. The study team also assessed reliability to confirm that items in the identified domains consistently measured the same construct. In addition, the study team applied other psychometric analysis methods to examine item functioning and differential item functioning across student subgroups. These analyses provided evidence of whether the KOT rating categories for each item were ordered correctly and whether there were any potential biases in how teachers rated student subgroups. Finally, the study team examined the proportion of the variation in the KOT's domain scores and item ratings at the classroom level to explore the extent to which domain scores and item ratings provide information about individual student abilities. The study identified valid and reliable approaches for scoring KOT item ratings, although not based on the developer's intended six-domain structure. Additional development and validation work is still needed to assess benchmarks for school readiness, determine whether particular items are biased for student subgroups, and examine the sources of classroom-level variations in scores.

Online Availability