Skip Navigation
Funding Opportunities | Search Funded Research Grants and Contracts

IES Grant

Title: Assessing Readers Struggling to Comprehend Multiple Sources of Information
Center: NCER Year: 2005
Principal Investigator: Lawless, Kimberly Awardee: University of Illinois, Chicago
Program: Reading and Writing      [Program Details]
Award Period: 4 years Award Amount: $1,562,428
Goal: Measurement Award Number: R305G050091

Co-Principal Investigator: Susan Goldman

Purpose: In upper elementary school science and social studies classes, students are often asked to read multiple texts in order to answer questions and prepare reports. However, most comprehension research examines how students derive meaning from a single text, and most comprehension assessment tools similarly ask students to read and answer questions about a single passage. The need to better understand how students make meaning as they read across multiple texts and for teachers to assess whether students are developing appropriate comprehension skills serves as the motivation for this research project. Its purpose is to develop a set of assessments that can diagnose the strategies and skills required for comprehension of multiple texts and identify where readers struggle in this complex but important task. At the conclusion of this project, the research team will have a rigorously documented taxonomy of comprehension strategies and skills central to multiple text comprehension and a set of assessments and indicators that differentiate among levels of proficiency, are useful in classroom contexts, and provide information for instructional decision making.

Setting: This project takes place in third-, fourth-, and fifth-grade classrooms in the public schools in a large midwestern city.

Population: Students participating in this project are economically and ethnically diverse. In Year 1, 54 students in six fifth-grade classrooms, and their teachers, are participating. In Year 2, approximately 225 students are participating in the quasi-experimental evaluation of the assessment components. In Years 3 and 4, both teachers and students are participating. Nine to 12 third-, fourth-, and fifth-grade teachers are participating in the evaluation of the appropriateness of the tasks for students. Approximately 450 third-, fourth-, and fifth-grade students are participating in the generalizability studies being carried out in the final two years of the project.

Research Design and Methods: Year 1 focuses on descriptive, micro-ethnographic research that documents students' activities with multiple sources of information as they carry out classroom research assignments in social studies or science project work. In Year 2, the researchers are developing tasks that are delivered in self-contained computer-based assessment environments and that describe student skills and knowledge of the components necessary for comprehension across texts. A series of quasi-experimental studies are being carried out to determine the most informative assessment tasks. In Years 3 and 4, the researchers are developing and administering various tasks to a range of students in rigorously designed field tests that are reviewed by classroom teachers. Finally, they are conducting field tests on larger samples of students to further examine issues of generalizability, reliability, validity, instructional usefulness, and feasibility of integration into classroom settings.

Data Analytic Strategy: Systematic qualitative analyses are being used on the Year 1 data. Generalizability studies will use analysis of variance.

Publications from this project:

Braasch, J. L. G., Lawless, K. A., Goldman, S. R., Manning, F. H., Gomez, K. W., and MacLeod, S. (2009). Evaluating Search Results: An Empirical Analysis of Middle School Students’ Use of Source Attributes to Select Useful Sources. Journal of Educational Computing Research, 41 (1), 63-82.

Goldman, S.R., Lawless, K.A., Pellegrino, J.W., Braasch, J.L.G., Manning, F.H., and Gomez, K. (2012). A Technology for Assessing Multiple Source Comprehension: An Essential Skill of the 21st Century. In M. Mayrath, J. Clarke-Midura, and D.H. Robinson (Eds.), Technology-Based Assessments for 21st Century Skills: Theoretical and Practical Implications From Modern Research (pp. 171-207). Charlotte, NC: Information Age Publishing.

Goldman, S. R., Lawless, K. A., Gomez, K. W., Braasch, J. B., MacLeod, S., & Manning, F. (2010). Literacy in the Digital World: Comprehending and Learning from Multiple Sources. In M G. McKeown and L. Kucan (Eds.), Threads of Coherence in Research on the Development of Reading Ability or Bringing Reading Researchers to Life. NY: Guilford.

Goldman, S.G., Ozuru, Y., Braasch, J.G., Manning, F.H., Lawless, K.A., Gomez, K.W., & Slanovits, M.J. (2011). Literacies for Learning: A Multiple Source Comprehension Illustration. To appear in N. Stein (Ed.), Development Science Goes to School. Taylor & Francis.

Goldman, S.R., and Scardamalia, M. (2013). Managing, Understanding, Applying, and Creating Knowledge in the Information Age: Next-Generation Challenges and Opportunities. Cognition and Instruction, 31 (2): 255-269. doi:10.1080/10824669.2013.773217

Hastings, P., Hughes, S., Magliano, J., Goldman, S., & Lawless, K. (2011). Text Categorization for Assessing Multiple Documents Integration, or John Henry Visits a Data Mine. In Proceedings of the 15th International Conference on Artificial Intelligence in Education, Lecture Notes in Artificial Intelligence 6738. G. Biswas, S. Bull, J. Kay, and A. Mitrovic, Eds. pp. 115-122. Springer-Verlag, Berlin.

Hastings, P., Hughes, S., Magliano, J.P., Goldman, S.R., & Lawless, K. (2012). Assessing the Use of Multiple Sources in Student Essays. Behavior Research Methods, 44 (3): 622-633. doi:10.3758/s13428-012-0214-0

Lawless, K., Braasch, J., Manning, F., Goldman, S., Ozuru, Y. & Gomez, K. (2008). Assessment of Digital Literacies: Initial Task Design and Piloting of a Source Selection Module. In G. Richards (Ed.), Proceedings of World Conference on E-Learning in Corporate, Government, Healthcare, and Higher Education 2008 (pp. 2880-2885). Chesapeake, VA: AACE.

Lawless, K. A., Goldman, S., Gomez, K., & Bertenthal, M. (2006). Initial Development of an Assessment of Digital Literacy Skills. Proceedings of the Cognition and Exploratory Learning in Digital Age Annual Conference, Barcelona, Spain.

Lawless, K.A., Goldman, S.R., Gomez, K., Manning, F., and Braasch, J.L. (2012). Assessing Multiple Source Comprehension Through Evidence Centered Design. In J.S. Sabatini, E. Albro, and T. O'Reilly (Eds.), Reaching an Understanding: Innovations in How We View Reading Assessment (pp. 3-17). Lanham, MD: Rowman and Littlefield Education.

Manning, F. H., Goldman, S. R., Ozuru, Y., Lawless, K. A., Gomez, K. , & Braasch, J. L. G. (2008). Students’ Analysis of Multiple Sources for Agreements and Disagreements. Proceedings of the Eighth International Conference of the Learning Sciences. Mahwah, NJ: Erlbaum.

Wiley, J., Goldman, S.R., Graesser, A.C., Sanchez, C.A., Ash, I.K., and Hemmerich, J.A. (2009). Source Evaluation, Comprehension, and Learning in Internet Science Inquiry Tasks. American Educational Research Journal, 46 (4): 1060–1106. doi:10.3102/0002831209333183