Co-Principal Investigator: Joseph Rios

Purpose: This project will develop a technology-enhanced observation system, the Tool to Observe Practices and Supports for Engagement (TOPSE), to simultaneously monitor and promote the use of essential provider interaction practices while measuring children's active engagement within early learning environments. Children's engagement with their environment is an essential ingredient for learning, and hence crucial to preschool child outcomes including school readiness. Although global classroom quality measures aid in creating foundations for learning, children with developmental delays and disabilities need educators to supplement and augment universally applied, high-quality classroom practices. Educators may need unique data and professional development to promote specific practices that serve the unique and often differentiated needs of young children within authentic daily routines. This project will address this need by designing a psychometrically sound observation and data-driven feedback system that can be used in a broad array of early learning environments to support educators' use of interaction practices that promote active engagement in learning for children with and without disabilities and improve children's outcomes.

Project Activities: An iterative process will be used over the course of the project to develop and validate the TOPSE. In Year 1, a blueprint specifying the factors and structure of the system will be created, followed by the development of items that will be repeatedly tested in classrooms representing a broad array of different learning environments. In Year 2, the first technology-based version of the system will be tested in a laboratory setting, followed by testing in authentic early learning environments to begin examination of administration, standardization, and validation. Years 3 and 4 will focus on refinement of administration and standards as well as validation of feasibility and use. This process will involve repeated observations within different classroom routines to refine and enhance the system's psychometric properties, and most importantly, the feasibility and use of the system as a tool for delivering data-based and targeted professional development.

Products: The primary product of this project will be a fully developed cloud-based observation system, the TOPSE, for use in a broad array of authentic early learning environments to guide and monitor the use of provider-child interaction practices that promote active engagement in learning for children with and without disabilities. The products will also include peer-reviewed publications and presentations.

Structured Abstract

Setting: Development of the TOPSE will begin in laboratory and controlled classroom settings before being further refined and tested in authentic early childhood classrooms that include preschoolers who are receiving early childhood special education (ECSE) services. Classrooms will be located in Minnesota and will represent a broad array of service delivery models (e.g., inclusive classrooms, dedicated ECSE), instructional paradigms (e.g., adult-directed, child-directed), and educator experience (e.g., team working together for years, new class team).

Sample: Over the course of 4 years, approximately 300 observers (novice and experienced educators), 200 preschool classrooms, and an estimated 1,500 children (ages 3-5) with and without disabilities and/or developmental delays will be involved in the research.

Assessment: The TOPSE is a technology-enhanced, cloud-based system that will support the implementation and ongoing monitoring of classroom interaction practices for promoting children's active engagement in natural routines. The TOPSE can be used for brief observation sessions across different types of classroom routines. Completion of the TOPSE will lead to automated graphic and written performance feedback within and across routines. This feedback will offer meaningful interpretations and recommendations about classroom practices in addition to the data summary and display. Unique to the TOPSE, performance feedback will explicitly link classroom practices to observed child engagement as a meaningful, proximal criterion by which to evaluate the impact of classroom practices.

Research Design and Methods: A multi-phase, iterative process will guide the development and validation of the TOPSE. Phase 1 (Blueprint) will occur during Year 1 and involve development of the overall plan/blueprint, content definition, and test specifications. Phase 2 (Production) will span Years 1 and 2 and include item development, test design and assembly, and test production. Phase 3 (Administration and Scoring) will span Years 2 through 4 and include test administration/observations, scoring responses, standard setting, score reporting, item banking, and technical reporting. Phase 4 (Validation) will occur throughout Years 1 through 4 of the project and will provide information on the TOPSE's feasibility, psychometric, and utility validation.

Control Condition: Due to the nature of the research design, there is no control condition.

Key Measures: The primary measure used in this project is the one being developed, the Tool for Observing Practices and Supporting Engagement (TOPSE). As part of measurement development of the TOPSE, children's observed engagement using momentary time sampling on a variable schedule will serve as the primary, most proximal criterion measure. To provide a preliminary evaluation of the constructs measured by TOPSE, the Classroom Assessment and Scoring System will be used in a subset of classrooms in Years 3 and 4. Further, to provide a preliminary evaluation of the child engagement ratings on the TOPSE relative to existing measures, the Social Skills Intervention Rating System as well as three tools for monitoring children's overall development and emerging academic skills (i.e., FAST's Developmental Milestones, earlyReading, and earlyMath) will be used with a subset of children in Years 3 and 4.

Data Analytic Strategy: The specific analytic approaches will vary in alignment with the intended outcomes of each phase of development. Across all phases of the iterative development process, the research team will use a combination of methods from Classical Test Theory (CTT), Generalizability Theory (GT), and Confirmatory Factor Analysis (CFA). CTT will be used to evaluate reliability, which includes internal consistency within and across factors and to estimate standard errors of measurement. Within the iterative process, this approach will also contribute to psychometric validation of the TOPSE. GT will be used in a series of generalizability and decision studies to refine measurement and scoring within and across types of settings, activities, and observers. Within the iterative process, this approach will also contribute to validation of feasibility and utility of the TOPSE. CFA will be used to test the hypothesis that the observed items represent the hypothesized factor structure and associated constructs. In addition, CFA will be used to scale the TOPSE scores. Within the iterative process, this approach will also contribute to validating the psychometric properties and utility of TOPSE. Although this project is primarily focused on development, preliminary analyses of construct, criterion, and predictive validity will be based on associating measures of interaction practices with classroom quality measures as well as associating children's engagement with measures of children's developmental skills.