Co-Principal Investigators: Kamata, Akihito; Sáez, Leilani; Nese, Rhonda

Purpose: The research team will develop and validate an automated scoring system to measure, unite, and scale the rate, accuracy, and prosody of oral reading fluency (ORF) for students in grades 2 to 4. Research has shown that prosody explains variance in reading comprehension beyond rate and accuracy; however, current ORF assessments neglect the measurement of prosody. This project can increase the reliability and validity of decisions made from ORF scores, resulting in better identification of students in need of reading interventions, and better evaluation of those interventions.

Project Activities: In Phase I of the project, the research team will conduct a study of human-rated prosody for a large sample of a data from a previously funded IES project, the Computerized Oral Reading Evaluation project (CORE) and apply a machine learning model to measure and score prosody. In Phase II, researchers will work to develop the automatic speech recognition (ASR) engine and compare it others to ensure it optimizes performance, integrate a pupil size change application to measure cognitive load, and develop the tablet application to administer the assessment. In Phase III, the research team will extend the work of CORE to produce a model of fluency that includes prosody as well as accuracy and rate and apply the model to a common scale across grades, providing an advantage for across-grades growth monitoring. In Phase IV, the team will evaluate assessment validity and will enhance the interpretability and usability of the assessment in a feasibility of use study. The team plans to engage in dissemination activities during each year of the grant, with Phase IV scheduled to be the most productive year for dissemination.

Products: The main products of this project include a novel approach to prosody measurement, a unified ORF score, and an entirely automated ORF assessment. The research team will develop a public-facing dissemination strategy using open access, strategic partnerships, a social media presence, and a project website. In addition, the team will present at practitioner and research-based conferences and publish research findings in peer reviewed journals.

Structured Abstract

Setting: This project will take place across suburban and town school districts in Oregon.

Sample: A primary study of the relation between fluency, and reading comprehension involves 600 students and teachers from 24 second- through fourth-grade classrooms. A secondary data analysis of the relation between fluency and reading comprehension involves about 3,835 students. Both samples encompass a wide range of reading proficiency levels, and socio-economic, ethnic, and language backgrounds.

Assessment: The research team will develop and validate an ORF measure that includes rate, accuracy, and prosody for students in grades 2 through 4 for screening and progress monitoring, extending the work of the previously funded IES project. The assessment is computer administered and scored to reduce lost instructional time and administration costs; offers a measure of prosody and a unified ORF score of accuracy, rate, and prosody to align with reading theory and provide educators with more meaningful reading information; and provides a vertical scale across grades to improve score reliability and progress monitoring accuracy.

Research Design and Methods: In Phase I, researchers will conduct a study of human-rated prosody for a large sample of CORE data and apply a machine learning model to measure and score prosody. In Phase II, the team will develop the automatic speech recognition (ASR) engine and compare it others to ensure it optimizes performance, integrate a pupil size change application to measure cognitive load, and develop the tablet application to administer the assessment. In Phase III, the research team extend the CORE binomial-lognormal joint factor model to produce a model of fluency that includes prosody as well as accuracy and rate and apply the model to their scaling framework by calibrating passage parameters on a common scale across grades, providing an advantage for across-grades growth monitoring. In Phase IV, the team evaluates assessment validity by analyzing the relation between accuracy, rate, prosody, their combined scale score, cognitive load, and reading comprehension. In addition, the research team will enhance the interpretability and usability of the assessment in a feasibility of use study.

Control Condition: Due to the nature of the design of this study, there is no control condition.

Key Measures: The research team will use the National Assessment of Educational Progress prosody scale for human-rated prosody, automatic speech recognition (ASR) to measure accuracy and rate and machine learning to estimate prosody. Criterion-related validity will be assessed using students' reading scores from the Smarter Balanced Assessment Consortium (SBAC) reading test and curriculum-based measurement comprehension measures. Students' cognitive load will be assessed using pupil size change. Interviews and observations will be used to assess feasibility of the assessment in classrooms.

Data Analytic Strategy: Researchers will employ a binomial-lognormal joint factor model to include prosody and produce a scale measure of fluency. They will use linear mixed-effects models to analyze the relation between comprehension and project estimates of accuracy, rate, prosody, and cognitive load, controlling for student demographic characteristics as appropriate.

Cost Analysis: The research team will conduct a comprehensive analysis of total economic costs using an “ingredients approach” to assess both total and net costs of the project ORF assessment (i.e., costs below/beyond the costs associated with traditional ORF assessments).

Related Projects: Measuring Oral Reading Fluency: Computerized Oral Reading Evaluation (CORE) (R305A140203)