Purpose: The purpose of this project is to develop a tablet-based application (app) that provides a comprehensive early literacy Response-to-Intervention (RTI) experience for children and supporting material for their educators. Based on already existing paper-and-pencil materials that include assessments for screening and identification (Individual Growth and Development Indicators, or IGDI), this technology-based app will also allow for the real-time collection and analysis of student data, resulting in timely RTI decision-making recommendations. Additionally, the system will leverage computer-adaptive testing to increase efficiency.

Project Activities: The research team will create digital versions of the existing paper materials. Researchers will also increase the bank of test items through an Item Response Theory (IRT) development process. They will also develop the testing protocol and the associated algorithms that will be used in the technology-driven RTI, with the goal of enabling a more efficient testing process via computer-adaptive testing. Once sufficient items and supporting materials (e.g., manuals for test administration, training documents) are completed, the team will conduct a pilot study to examine how well a child could use the app alone (i.e., self-administer) compared to when a teacher administers it. A second pilot study will compare three learning conditions, each using the app to varying degrees. All conditions will employ the app for assessment purposes over the course of the academic year, both for initial placement (in the fall) and progress monitoring (in the winter and spring). One group will also use the to-be-developed technology throughout the year as an instructional tool whereas another group will use the current paper-and-pencil version as an instructional tool. The last group will use typical practices of instruction (i.e., business as usual). Additional activities include the creation of a website and development of the cloud infrastructure for data storage and facilitating reporting.

Products: Products from this study will be a mobile, tablet-based app to facilitate early childhood RTI. The app will include computer-adaptive assessments, real-time supports for instructional decision making, learning materials, progress monitoring, and report production. Peer reviewed publications and conference presentations will also be produced.

Structured Abstract

Setting: This project will take place in three early childhood learning centers from both urban and suburban areas in Minnesota.

Sample: Participants for this project include approximately 60 teachers and 670 4-year old English-speaking children, including both typically developing and those at-risk.

Intervention: The intervention, consisting of both instructional and assessment components, will employ psychometrically robust measures specifically designed both to identify students for tier level recommendations during screening and monitor their progress in mastering components of early literacy during use of the intervention. The assessment component will employ robust computer-adaptive testing in order to reduce the testing burden required to produce reliable and valid data. The instructional component will support decision making based on real-time data gathered through live interaction and offer supporting data interpretation for instructional suggestions. Additionally, the system will generate reports for individuals, groups, classes and districts.

Research Design and Methods: Initial project activities will include creating digital versions of extant paper-and-pencil IGDIs, and developing additional assessment items to increase the current test bank. The research team will assess feasibility, usability, and fidelity of implementation of the intervention through surveys and focus groups of teachers during field-testing, employing techniques such as semi-structured interviews, seeking opinions and suggestions for improvements, think-aloud activities to examine teachers’ experience with the intervention, and conducting observations. In Pilot Study 1, the research team will compare two delivery modes of the intervention in an experimental design. Half of the participating students will self-administer the intervention, and the other half will receive the intervention via teacher administration. In Pilot Study 2, the researchers will carry out a randomized trial, with groups of teachers (and their students) assigned to use one of three instructional conditions: the to-be-developed intervention as an instructional tool, the current paper-and-pencil version, or business-as-usual.

Control Condition: In the second pilot study, teachers will follow their standard instructional practices (i.e., business-as-usual), although the intervention will be used for assessment purposes. The first pilot study does not have a control condition.

Key Measures: The key measures of student early literacy performance will focus on oral language, phonological awareness, alphabet recognition, and comprehension, as assessed by measures developed both by the Center for Response to Intervention in Early Childhood and the Preschool Early Literacy Indicators. Teacher self-efficacy will be measured through the Teacher’s Sense of Self-Efficacy Scale.

Data Analytic Strategy: During development of new assessment items, the team will employ Rasch modeling in order to locate item difficulty and to compute student scores for the CAT. When testing the intervention, the research team will use descriptive summary statistics, analysis of variance, and hierarchical linear models of key measures for both student and teacher performance, and correlations between teacher behaviors and student outcomes.

Products and Publications

Will, K. K., McConnell, S. R., Elmquist, M., Lease, E. M., & Wackerle-Hollman, A. (2019). Meeting in the Middle: Future Directions for Researchers to Support Educators' Assessment Literacy and Data-Based Decision Making. In Frontiers in Education, Vol. 4(106): 1–8.