Co-Principal Investigator: Durán, Lillian

Purpose: The purpose of this project is to develop a new tablet-based measure in English and Spanish, Computer-Adaptive Storybook Assessment (CASA), to monitor language and literacy progress in preschoolers with disabilities or at risk for later language difficulties. Language and early literacy development are key hallmarks for later academic and social success. Studies demonstrate a relationship between language and early literacy performance during preschool across three domains (oral language, phonological awareness, and alphabet knowledge) and later reading success. There is additional evidence that many children with or at risk for disabilities lag further behind their peers, demonstrating poor reading performance in later grades. Instructionally relevant assessments are needed to support differentiated learning for preschoolers with disabilities or at risk for reading failure to help teachers apply appropriate evidence-based intervention and instruction. To meet this need, this project will develop and validate a gamified application that leads students through stories with embedded assessment items that are tailored to the child's language (English or Spanish) and are culturally relevant to assess abilities in oral language, phonological awareness, and alphabet knowledge.

Project activities:This project will use a variety of assessment methods to develop and validate a computerized adaptive testing assessment, CASA, to assess early language and literacy skills in low performing, high-risk children who speak English or Spanish. The project will also estimate the costs of using the assessment through a cost analysis.

Products: The primary product of this project is a fully developed, validated, and functional app that administers computer-adaptive testing to monitor progress in early language and literacy for English- or Spanish-speaking preschool children, including those who have disabilities, are dual-language learners, or are otherwise at risk for later reading failure. The project will result in peer-reviewed publications and presentations as well as additional dissemination products that reach education stakeholders, such as practitioners and policymakers.

Structured Abstract

Setting: This research will take place in preschool classrooms across various regions of the country, including Oregon, California, Utah, Minnesota, and Puerto Rico.

Sample: This project will include 800 3- to 5-year-old children who are at risk for later reading difficulties, including (a) children with developmental language disorders, (b) children whose first language is Spanish (monolingual or bilingual), and (c) children identified as at risk through individual performance or demographic factors. To inform the project at various stages, the study will also include up to 80 teachers, a panel of five experts (in preschool language development, speech and language disorders, Spanish language development, and early literacy skill development), and five preschool administrators.

Assessment: This project will develop CASA, an engaging and instructionally relevant progress monitoring storybook assessment delivered via tablet application for preschool children who have low language and early literacy skills and are at risk for later reading disabilities and challenges. All items will be scaled for use in a computer-adaptive testing engine so that for each scene in the story, children interact with different items based on their current ability level. The assessment is expected to take approximately 15 minutes and measure three language domains (oral language, phonological awareness, and alphabet knowledge) in English and Spanish. CASA will provide rigorous, actionable information for teachers to promote language and early literacy skills for their high-risk students who speak English or Spanish.

Research Design and Methods: This study will proceed in five phases that iteratively engage an agile application development process to develop CASA. In the first phase, the research team will develop story content in English and Spanish and assessment items that will be integrated into the storybook. In the second phase, a prototype of the assessment will be iteratively developed and tested. A field test with teachers and children in preschool settings will be used in the third phase to test how the items work and merge into constructs (factor structure) and the full suite of items in Spanish and English will be developed. In the fourth phase, items will be integrated into the computer-adaptive administration platform. In the final phase, the application will be revised to incorporate feedback from previous stages to produce a full computer-adaptive storybook experience for children with complementary teacher administration manuals and supports. During this phase, CASA will be used across the academic year to collect data to model growth and examine validity against criterion measures administered at the beginning and end of the academic year.

Control Condition: Due to the nature of the research design, there is no control condition.

Key Measures:  To validate CASA, the research team will use a variety of standardized, norm-referenced criterion measures, including the Individual Growth and Development Indicators in English and Spanish, the Quick Interactive Language Screener in English and Spanish, the MacArthur Bates-Communicative Development Inventories III and MacArthur Inventarios del Desarrollo de Hablidades Communicatives, and the Bilingual Assessment of Phonological Sensitivity. There will also be investigator-developed measures to understand implementation, including focus group feedback forms, fidelity of implementation checklists, and feasibility and utility surveys. Finally, the team will collect a variety of data from the families of children who participate and their teachers using a family survey, the Language Environment Evaluation Report, a classroom survey, and teacher demographic data.

Data Analytic Strategy: Rasch modeling will be used to scale item difficulties. The team will also conduct confirmatory factor analysis to evaluate how well items measure a single underlying construct for each domain, Pearson's r to correlate CASA scores with criterion measures to test for validity, and HLM to model growth and assess sensitivity to growth.

Cost Analysis: Once CASA is completed, the ingredients method will be used to collect data for a cost analysis. The analysis will be conducted using the CostOut tool to estimate individual costs. Using this information, the team will prepare three scenarios that examine costs related to the product, materials, assessor training, school time, teacher or assessor time, and other factors.