Grant Closed

Assessing Readers Struggling to Comprehend Multiple Sources of Information

NCER

Program: Education Research Grants

Program topic(s): Literacy

Award amount: $1,562,428

Principal investigator: Kimberly Lawless

Awardee:

University of Illinois, Chicago

Year: 2005

Project type:

Measurement

Award number: R305G050091

Purpose

In this project, the researchers proposed to develop a set of assessments that could diagnose the strategies and skills required for comprehension of multiple texts. The purpose of this assessment was to identify what causes readers to struggle with this complex but important task. In the early 2000s, most comprehension research examined how students derive meaning from a single text, so there was a need to better understand how students make meaning as they read across multiple texts and for teachers to assess whether students were developing appropriate comprehension skills. At the conclusion of the project, the research aimed to have documented a taxonomy of comprehension strategies and skills central to multiple text comprehension and developed a set of assessments and indicators that differentiate among levels of proficiency, are useful in classroom contexts, and provide information for instructional decision-making.

Structured Abstract

Setting

This project takes place in third-, fourth-, and fifth-grade classrooms in the public schools in a large midwestern city.

Sample

Students participating in this project are economically and ethnically diverse. In phase 1, 54students in 6 grade 5 classrooms and their teachers are participating. In phase 2, approximately 225 students are participating in the quasi-experimental evaluation of the assessment components. In phase 3 and 4, both teachers and students are participating. Approximately 9 to 12 grade 3, 5, and 5 teachers are participating in the evaluation of the appropriateness of the tasks for students. Approximately 450 grade 3, 4, and 5 students are participating in the generalizability studies being carried out in the final 2 years of the project.

Research design and methods

Phase 1 focuses on descriptive, micro-ethnographic research that documents students' activities with multiple sources of information as they carry out classroom research assignments in social studies or science project work. In phase 2, the researchers are developing tasks that are delivered in self-contained computer-based assessment environments and that describe student skills and knowledge of the components necessary for comprehension across texts. A series of quasi-experimental studies are being carried out to determine the most informative assessment tasks. In phases 3 and 4, the researchers are developing and administering various tasks to a range of students in rigorously designed field tests that are reviewed by classroom teachers. Finally, they are conducting field tests on larger samples of students to further examine issues of generalizability, reliability, validity, instructional usefulness, and feasibility of integration into classroom settings.

Data analytic strategy

Systematic qualitative analyses are being used on the phase 1 data. Generalizability studies will use analysis of variance.

People and institutions involved

IES program contact(s)

Elizabeth Albro

Commissioner of Education Research

NCER

Products and publications

ERIC Citations: Find available citations in ERIC for this award here.

Select Publications:

Book chapters

Goldman, S.R., Lawless, K.A., Gomez, K.W., Braasch, J.L.G., MacLeod, S., and Manning, F. (2010). Literacy in the Digital World: Comprehending and Learning From Multiple Sources. In M.C. McKeown, and L. Kucan (Eds.), Bringing Reading Research to Life (pp. 257-284). New York: GuilfordPublications.

Goldman, S.R., Lawless, K.A., Pellegrino, J.W., Braasch, J.L.G., Manning, F.H., and Gomez, K. (2012). A Technology for Assessing Multiple Source Comprehension: An Essential Skill of the 21st Century. In M. Mayrath, J. Clarke-Midura, and D.H. Robinson (Eds.), Technology-Based Assessments for 21st Century Skills: Theoretical and Practical Implications From Modern Research(pp. 171-207). Charlotte, NC: Information Age Publishing.

Goldman, S.R., Ozuru, Y., Braasch, J., Manning, F., Lawless, K. Gomez, K., and Slanovits, M. (2011). Literacies for Learning: A Multiple Source Comprehension Illustration. In N.L. Stein, and S.W. Raudenbush (Eds.), Developmental Science Goes to School: Implications for Policy and Practice(pp. 30-44). New York: Routledge.

Lawless, K.A., Goldman, S.R., Gomez, K., Manning, F., and Braasch, J.L. (2012). Assessing Multiple Source Comprehension Through Evidence Centered Design. In J.S. Sabatini, E. Albro, and T. O'Reilly (Eds.), Reaching an Understanding: Innovations in How We View Reading Assessment (pp. 3-17). Lanham, MD: Rowman and Littlefield Education.

Journal articles

Braasch, J.L.G., Lawless, K.A., Goldman, S.R., Manning, F., Gomez, K.W., and MacLeod, S. (2009). Evaluating Search Results: An Empirical Analysis of Middle School Students' Use of Source Attributes to Select Useful Sources. Journal of Educational Computing Research, 41(1): 63-82.

Goldman, S.R., and Scardamalia, M. (2013). Managing, Understanding, Applying, and Creating Knowledge in the Information Age: Next-Generation Challenges and Opportunities. Cognition and Instruction, 31(2): 255-269.

Hastings, P., Hughes, S., Magliano, J.P., Goldman, S.R., and Lawless, K. (2012). Assessing the Use of Multiple Sources in Student Essays. Behavior Research Methods, 44(3): 622-633.

Wiley, J., Goldman, S.R., Graesser, A.C., Sanchez, C.A., Ash, I.K., and Hemmerich, J.A. (2009). Source Evaluation, Comprehension, and Learning in Internet Science Inquiry Tasks. American Educational Research Journal, 46(4): 1060-1106.

Proceedings

Hastings, P., Hughes, S., Magliano, J., Goldman, S., and Lawless, K. (2011). Text Categorization for Assessing Multiple Documents Integration, or John Henry Visits a Data Mine. In Proceedings of the 15th International Conference on Artificial Intelligence in Education (pp. 115-122). Berlin: Springer-Verlag.

Lawless, K.A., Braasch, J.L.G., Manning, F.H., Goldman, S.R., Ozuru, Y., and Gomez, K.W. (2008). Assessment of Digital Literacies: Initial Task Design and Piloting of a Source Selection Module. In Proceedings of World Conference on E- Learning in Corporate, Government, Healthcare, and Higher Education 2008 (pp. 2880-2885). Chesapeake, VA: AACE.

Lawless, K.A., Goldman, S., Gomez, K., and Bertenthal, M. (2006). Developing an Assessment of Digital Literacy Skills: Definition of the Student Model. In Proceedings of the IADIS International Conference on Cognition and Exploratory Learning in Digital Age (CELDA 2006) (pp. 433-434). Lisbon, Portugal: International Association for Development of the Information Society (IADIS).

Manning, F.H., Goldman, S.R., Ozuru, Y., Lawless, K.A., Gomez, K. , and Braasch, J.L.G. (2008). Students' Analysis of Multiple Sources for Agreements and Disagreements. In International Perspectives in the Learning Sciences: Creating a Learning World: Proceedings of the Eighth International Conference of the Learning Sciences - ICLS 2008 (pp. 19-26). Utrecht, The Netherlands: International Conference of the Learning Sciences.

Supplemental information

Co-Principal Investigator(s): Goldman, Susan

Questions about this project?

To answer additional questions about this project or provide feedback, please contact the program officer.

Assessing Readers Struggling to Comprehend Multiple Sources of Information

Purpose

Structured Abstract

Setting

Sample

Research design and methods

Data analytic strategy

People and institutions involved

IES program contact(s)

Elizabeth Albro

Products and publications

Supplemental information

Questions about this project?

Questions about this project?

You may also like

Data Science Methods for Digital Learning Platform...

Meta-Analysis Training Institute (MATI)

Bayesian Longitudinal Data Modeling in Education S...