Project Website: http://www.cognitivetoybox.com

Video Demonstration of the Phase I Prototype: https://youtu.be/uItgvNcDg

Purpose: This project will fully develop and evaluate a hybrid observation and game-based assessment platform for early childhood educators. One core component of this project is the web-based dashboard and navigation system. Teachers use this component to translate child-directed assessment game data on skills, including early literacy and numeracy, into actionable insights for instructional adjustments and individualized support. Early childhood education programs require accurate, ongoing assessment to reach their school readiness goals. Most programs mandate child-level assessment, and programs are given the choice between observation-based or direct assessment. Observation-based assessment relies on teacher ratings that can be burdensome, subjective, and difficult to calibrate across different classrooms. Moreover, the lack of standardization makes it difficult for programs to find ways to improve.

Project Activities: During Phase I in 2019, the team developed a prototype dashboard that early-childhood educators used to review assessment data collected from the observation-based and game-based tools and track progress of individual children towards school readiness. At the end of Phase I, in a pilot study with 5 early childhood educators and 100 children, the researchers demonstrated that all of the educators successfully integrated the prototype with no disruption to their daily practices, that the assessments matched educators own qualitative understanding of children's abilities, and that the prototype eased monitoring of child progress and helped identify areas where children need instructional support.

In Phase II, the team will iteratively refine and then fully develop the dashboard, the early childhood formative assessments, and the navigation system so that the assessment data is actionable for early-childhood educators. During this process, validity and reliability of the assessments will be examined. After development is complete, the team will carry out a small-scale randomized controlled study with 16 pre-Kindergarten classrooms and 400 students (25 per class) to examine the usability and feasibility, fidelity of implementation, and the promise of the dashboard to inform teacher insights and guide instruction, and improve children's school readiness. Half of the classes will be randomly assigned to use the intervention and the dashboard for 12 weeks and the other half assigned to a business-as-usual group. The researchers will use standardized measures including the Phonological Awareness Literacy Screening and the Individual Growth and Development Indicators of Early Numeracy to measure student achievement. Given the cluster randomized design and inherent nested structure, a hierarchical linear model will be employed to estimate the impact of the dashboard and navigation system on student numeracy and literacy outcomes. Costs will be calculated based on the actual per student and class wide implementation costs.

Product: This project will fully develop an early childhood holistic assessment platform that combines observation and direct (game-based) tools to assess the school readiness of individual children. This approach will reduce the assessment burden for teachers and provide usable data to improve instruction in real-time. The games are intended for low-stakes assessment – children interact with the adaptive, touchscreen games for 5 minutes per week—and teachers use the automatically generated reports to adjust instruction.

Related IES Project: Navigation System for Early Childhood Assessment and Instruction (91990019C0021)