Skip Navigation
Funding Opportunities | Search Funded Research Grants and Contracts

IES Grant

Title: Assessing Reading Comprehension with Verbal Protocols and Latent Semantic Analysis
Center: NCER Year: 2004
Principal Investigator: Magliano, Joseph Awardee: Northern Illinois University
Program: Reading and Writing      [Program Details]
Award Period: 4 years Award Amount: $1,560,506
Goal: Measurement Award Number: R305G040055

Co-Principal Investigator: Keith Millis

Purpose: Many college students have great difficulty understanding their textbooks and are at risk of receiving poor grades or being academically dismissed. In many cases, the reading problem does not arise from a deficient in lower-language skills, such as word decoding, but from having a low standard of coherence. Readers who adopt a low standard of coherence read passively, finding a sparse and perhaps incoherent representation acceptable. Readers who have a high standard of coherence employ an effortful search for meaning in which they construct a coherent understanding of what the text is about. These readers use a variety of reading strategies - processes that the reader uses to understand the text. The proposed project will develop and test a new automated on-line reading strategy assessment tool called the Reading Strategy Assessment Tool (R-SAT).

Assessment: R-SAT will identify basic reading strategies and the extent to which the reader is actively engaged. R-SAT provides two very important advantages over existing reading tests that rely on a multiple-choice format. The first is that R-SAT provides an assessment of the level of coherence and reading strategies employed by the student, and the second is that it measures comprehension during reading. Because multiple-choice tests measure comprehension after reading, they are subject to test-taking strategies and processes that adversely affect the validity of the instrument. For example, a student may first read the questions before reading the test passage, or a listed answer to a question might actually trigger the correct answer.

Population: The proposed research will study comprehension skills in a population of college freshman and sophomores attending Northern Illinois University (NIU). NIU has a diverse population of students of vary academic abilities and provides an ideal population to study individual differences in reading comprehension abilities.

Research Design and Methods: R-SAT builds upon innovative recent research that has used various computational approaches to analyze students' written or verbal comments solicited at critical points during the reading process. It has the potential to revolutionize how reading is assessed in the future. The computational approach uses latent semantic analysis in tandem with recent advancements in word-matching algorithms. R-SAT is based on recent published research that has shown that these computational approaches are able to classify reading strategies on par with trained human judges, and that scores based upon them predict measures of comprehension (recall, question answering). There will be three phases to the proposed research. Phase 1 will involve testing among methods of collecting and evaluating verbal protocols, identifying texts and sentences, collecting representative answers/verbal protocols (for norming purposes), and initial validity testing. The second phase involves implementing R-SAT as a web-based system, establishing its reliability and validity, testing among text formats, identifying its limitations, and comparing R-SAT to standard tests of comprehension. In the third phase, R-SAT will be incorporated into an existing web-based reading trainer (iSTART) and evaluated in a college course taken by freshmen and sophomores majoring in a wide variety of topics.

Publications from this project:

Britt, M.A., Wiemer, K., Millis, K.K., Magliano, J.P., Wallace, P., & Hastings, P. (2012). Understanding and Reasoning with Text. To appear in P. McCarthy & C. Boonthum (Eds), Applied Natural Language Processing and Content Analysis: Identification, Investigation, and Resolution Jefferson. IGI Global Publisher.

Gilliam, S., Magliano, J.P., Millis, K. K., Levinstein, I., & Boonthum, C. (2007). Assessing the Format of the Presentation of Text in Developing a Reading Strategy Assessment Tool (RSAT). Behavior Research Methods, Instruments, & Computers, 39, 199-204.

Kurby, C.A., Magliano, J.P., Dandotkar, S., Woehrle, J., Gilliam, S., and McNamara, D.S. (2012). Changing How Students Process and Comprehend Texts With Computer-Based Self-Explanation Training. Journal Of Educational Computing Research, 47 (4): 429–459.

Magliano, J. P., Millis, K.K., The RSAT Development Team, Levinstein, I., & Boonthum, C (2011). Assessing Comprehension During Reading with the Reading Strategy Assessment Tool (RSAT). Metacognition and Learning, 6 (2) 131-154.

Magliano, J.P., Millis, K.K., Ozurur, Y., & McNamara, D.S. (2007). A Multidimensional Framework to Evaluate Assessment Tools. In D.S. McNamara (Ed.) Reading comprehension strategies: Theories, Interventions, and Technologies, (pp. 107-136). Mahwah, NJ: Erlbaum.

Magliano, J.P., & Perry, P.J. (2008). Individual Differences in Reading Proficiencies and Comprehension. In N.J. Salkind (Ed). Encyclopedia of Educational Psychology, Vol. 2, 511-517. Thousand Oaks, CA: Sage Publications, Inc.

Malladi, R. Levinstein, I.B., Boonthum, C., & Magliano, J. P. (2010). Summarization: Constructing an Ideal Summary and Evaluating a Student’s Summary using LSA. In the Proceedings of the 23rd International Florida Artificial Intelligence Research Society (FLAIRS) Conference, 295-296. Menlo Park, CA: The AAAI Press.

McNamara, D. S., & Magliano, J. P. (2009). Self-explanation and Metacognition: The Dynamics of Reading. In D. J. Hacker, J. Dunlosky, & A. C. Graesser, (Eds.), Handbook of Metacognition in Education (pp. 60-81). Mahwah, NJ: Lawrence Erlbaum and Associates.

McNamara, D. S., & Magliano, J. P. (2009). Towards a Comprehensive Model of Comprehension. In B. Ross (Ed.), The Psychology of Learning and Motivation, Vol. 51, (pp. 297-384), Burlington: Academic Press.

Millis, K.K., Magliano, J.P., & Todaro, S. (2006). Measuring Discourse-Level Processes with Verbal Protocols and Latent Semantic Analysis. Scientific Studies of Reading, 10, 251-283.

Millis, K.K., Magliano, J.P., Todaro, S., and McNamara, D.S. (2007). Assessing and Improving Comprehension With Latent Semantic Analysis. In T. Landauer, D.S. McNamara, S. Dennis, and W. Kintsch (Eds.), Handbook of Latent Semantic Analysis (pp. 207-226). Mahwah, NJ: Erlbaum

Muñoz, B., Magliano, J.P., Sheridan, R., & McNamara, D.S. (2006). Typing Versus Thinking Aloud When Reading: Implications for Computer-Based Assessment and Training Tools. Behavior Research Methods, Instruments, & Computers, 38, 211-217.

Mylavarapu, S., Levinstein, I.B., Boonthum, C., Magliano, J.P., & Millis, K.K. (2010). Enhancing Protocol Evaluation Through Semantic Modification of Benchmarks. In the Proceedings of the 23rd International Florida Artificial Intelligence Research Society (FLAIRS) Conference, 297-298. Menlo Park, CA: The AAAI Press.