Co-Principal Investigators: Kendeou, Panayiota; Connor, Carol M.

Purpose: In this project, researchers will develop and test an on-line tool that provides automated support to students in grade 3 and 4 on reading comprehension strategies. The ability to comprehend while reading high-quality texts is a strong predictor of success in the classroom and workplace, yet many students struggle to adequately develop this skill. According to the 2011 National Assessment of Educational Progress, 21% of high school seniors in the U.S. did not achieve basic proficiency in reading and only 3% performed well enough to be considered advanced readers.

Project Activities: In Years 1 to 3, the researchers will conduct a series of design-based implementation research studies to develop and refine the tool with feedback from students, teachers, and researchers. The studies will focus on creating and testing a user interface with an underlying natural language processing engine combined with automated speech recognition to enable immediate summative and formative feedback. In Year 4, the researchers will conduct an underpowered efficacy trial to measure the promise of the tool to produce results to improve reading.

Products: Researchers will develop iSTART-Early, an intelligent tutoring system that provides automated instruction to students in grades 3 and 4 on higher-order reading comprehension strategies. An interface will support teachers in administering essay assignments, which can then be either automatically scored or hand-graded by scaffolded rubrics. Students will use this tool to support reading and research findings from this project will advance understanding of how educational technologies can improve learning from texts.

Structured Abstract

Setting: The iterative studies will take place in cooperating grade schools in Minnesota, with a diverse set of student participants in terms of ethnicity, culture, and language across rural and suburban settings.

Sample: Research in Years 1 to 3 will involve focus groups and usability studies with small groups of upper elementary grade students and teachers. Participants in the Year 4 pilot study will include approximately 800 students in grades 3 and 4, and 90 teachers in grades 4 and 5.

Intervention: Researchers will develop iSTART-Early to provide personalized, interactive, game-based strategy instruction and practice on comprehension strategies, that is, comprehension monitoring, paraphrasing, inferencing, question asking, explanation, and summarization, with grade appropriate informational texts. Natural language processing combined with automated speech recognition technologies will allow immediate summative and formative feedback. A teacher interface will allow teachers to assign texts and monitor students' performance so that they can provide additional support and feedback when necessary, creating blended-learning opportunities.

Research Design and Methods: To develop the tool, the researchers will use a design-based implementation procedure whereby iterations of research and development will occur until feasibility, usability, and learning aims are met. The steps in the technological development will focus on developing the interface, the text library, lessons, and games; and computational algorithms using natural language processing to drive feedback. After development is complete, in Year 4 the team will conduct an underpowered randomized controlled trial to explore implementation in authentic classrooms and determine whether the tool shows promise of improving students' comprehension strategy use. The 14-week study will be a pretest-posttest experimental design and will include delayed treatment control classrooms as the comparison. Researchers will collect multiple sources of data, including surveys and log data to assess the impact of the tool on student reading within the real-world constraints and needs of the classroom.

Control Condition: Researchers will observe business-as-usual comprehension instruction in the control classrooms to identify how it compares to iSTART-Early.

Key Measures: In the randomized control trial pilot study, the researchers will use standardized reading assessments, including the Reading Comprehension Strategies Test Comprehension Monitoring Task, the Reading Comprehension Strategies Test, the Gates-MacGinitie Reading Test (GMRT), the Woodcock-Johnson III Tests of Achievement, and the Vocabulary test of the GMRT. Another set of measures will be used to assess the usability, feasibility, and fidelity of the intervention, including questionnaires, surveys, and logs.

Data Analytic Strategies: The team will use mixed methods, including qualitative analytic methods to analyze transcripts of design team meetings, teacher and student interviews, and classroom observation. Researchers will use hierarchical linear and a mixed-effects models to determine the promise of the intervention and moderating effects.

Related IES Project: iSTART: Interactive Strategy Trainer for Active Reading and Thinking (R305G040046)