Project Activities
In Phase I of the project, the research team will conduct a study of human-rated prosody for a large sample of a data from a previously funded IES project, the Computerized Oral Reading Evaluation project (CORE) and apply a machine learning model to measure and score prosody. In Phase II, researchers will work to develop the automatic speech recognition (ASR) engine and compare it others to ensure it optimizes performance, integrate a pupil size change application to measure cognitive load, and develop the tablet application to administer the assessment. In Phase III, the research team will extend the work of CORE to produce a model of fluency that includes prosody as well as accuracy and rate and apply the model to a common scale across grades, providing an advantage for across-grades growth monitoring. In Phase IV, the team will evaluate assessment validity and will enhance the interpretability and usability of the assessment in a feasibility of use study. The team plans to engage in dissemination activities during each year of the grant, with Phase IV scheduled to be the most productive year for dissemination.
Structured Abstract
Setting
This project will take place across suburban and town school districts in Oregon.
Sample
A primary study of the relation between fluency, and reading comprehension involves 600 students and teachers from 24 second- through fourth-grade classrooms. A secondary data analysis of the relation between fluency and reading comprehension involves about 3,835 students. Both samples encompass a wide range of reading proficiency levels, and socio-economic, ethnic, and language backgrounds.
Assessment
The research team will develop and validate an ORF measure that includes rate, accuracy, and prosody for students in grades 2 through 4 for screening and progress monitoring, extending the work of the previously funded IES project. The assessment is computer administered and scored to reduce lost instructional time and administration costs; offers a measure of prosody and a unified ORF score of accuracy, rate, and prosody to align with reading theory and provide educators with more meaningful reading information; and provides a vertical scale across grades to improve score reliability and progress monitoring accuracy.
Research design and methods
In Phase I, researchers will conduct a study of human-rated prosody for a large sample of CORE data and apply a machine learning model to measure and score prosody. In Phase II, the team will develop the automatic speech recognition (ASR) engine and compare it others to ensure it optimizes performance, integrate a pupil size change application to measure cognitive load, and develop the tablet application to administer the assessment. In Phase III, the research team extend the CORE binomial-lognormal joint factor model to produce a model of fluency that includes prosody as well as accuracy and rate and apply the model to their scaling framework by calibrating passage parameters on a common scale across grades, providing an advantage for across-grades growth monitoring. In Phase IV, the team evaluates assessment validity by analyzing the relation between accuracy, rate, prosody, their combined scale score, cognitive load, and reading comprehension. In addition, the research team will enhance the interpretability and usability of the assessment in a feasibility of use study.
Control condition
Due to the nature of the design of this study, there is no control condition.
Key measures
The research team will use the National Assessment of Educational Progress prosody scale for human-rated prosody, automatic speech recognition (ASR) to measure accuracy and rate and machine learning to estimate prosody. Criterion-related validity will be assessed using students' reading scores from the Smarter Balanced Assessment Consortium (SBAC) reading test and curriculum-based measurement comprehension measures. Students' cognitive load will be assessed using pupil size change. Interviews and observations will be used to assess feasibility of the assessment in classrooms.
Data analytic strategy
Researchers will employ a binomial-lognormal joint factor model to include prosody and produce a scale measure of fluency. They will use linear mixed-effects models to analyze the relation between comprehension and project estimates of accuracy, rate, prosody, and cognitive load, controlling for student demographic characteristics as appropriate.
Cost analysis strategy
The research team will conduct a comprehensive analysis of total economic costs using an "ingredients approach" to assess both total and net costs of the project ORF assessment (i.e., costs below/beyond the costs associated with traditional ORF assessments).
People and institutions involved
IES program contact(s)
Products and publications
Products: The main products of this project include a novel approach to prosody measurement, a unified ORF score, and an entirely automated ORF assessment. The research team will develop a public-facing dissemination strategy using open access, strategic partnerships, a social media presence, and a project website. In addition, the team will present at practitioner and research-based conferences and publish research findings in peer reviewed journals.
ERIC Citations: Find available citations in ERIC for this award here.
Related projects
Supplemental information
Co-Principal Investigators: Kamata, Akihito; Sáez, Leilani; Nese, Rhonda
Questions about this project?
To answer additional questions about this project or provide feedback, please contact the program officer.
