Skip Navigation
Funding Opportunities | Search Funded Research Grants and Contracts

IES Grant

Title: Assessing Online Reading Comprehension: The ORCA Project
Center: NCER Year: 2009
Principal Investigator: Leu, Donald Awardee: University of Connecticut
Program: Literacy      [Program Details]
Award Period: 4 years Award Amount: $2,813,127
Type: Measurement Award Number: R305A090608

Co-Principal Investigators: Jonna Kulikowich (Penn State University) and Julie Coiro (University of Rhode Island)

Purpose: During the twenty-first century, the internet has become a major force in transmitting information and has, hence, become an important learning environment, especially for reading comprehension. However, the reading comprehension skills that the internet requires are not necessarily the same as are required by more traditional, printed texts, and the American school system is not yet in a position to effectively teach these skill. Part of the reason for this is that we lack valid, reliable, and practical assessments of online reading comprehension that would inform instruction and help students become better online readers. The proposed research will address this issue by creating the Online Reading Comprehension Assessment (ORCA), which will allow stakeholders to assess whether students are acquiring the skills necessary for online reading comprehension.

Project Activities: The proposed Online Reading Comprehension Assessment (ORCA) Project will build upon previous assessments that were piloted with smaller populations. The team intends to use a systematic, iterative process to accomplish four major goals: 1) develop three different types of valid and reliable assessments to measure online reading comprehension; 2) evaluate the internal assessment characteristics for each instrument type to inform decisions about which is most useful and practical for schools; 3) evaluate the extent to which performance on each format is associated with four measures of contextual validity; and 4) both the utility and practicality of each instrument in the eyes of key decision makers: teachers, school administrators, and Chief State School Officers. The materials will be pilot tested during the initial development and then be evaluated in a large-scale assessment experiment. An interactive, online site will be developed to both disseminate ongoing results and invite additional insights and suggestions from the research community as the work unfolds.

Products: The researchers will develop a series of online reading comprehension assessments and an interactive, online site to both disseminate ongoing results and invite additional insights and suggestions from the research community as the work unfolds.

Structured Abstract

Setting: The study will take place in Maine and Connecticut. The ORCA Project is a partnership with these two states and with the Regional Educational Lab for New England and Islands.

Population: Participants will include 7th grade students.

Intervention: The research team will develop and evaluate multiple assessment formats for the measurement of online reading comprehension. These assessments will focus on science material such as human biology. The team has already developed and used a wide range of assessment formats from Open ORCAs to Multiple Choice ORCAs, and they propose to refine both the open and multiple-choice assessment approaches so that they can maximize the potential of each format. In addition, they will develop and evaluate a Closed ORCA, where a simulation of the Internet is created in a multimedia program with fixed informational boundaries that will not change over time. The built-in features of this Closed ORCA environment will permit the researchers to define rich online reading comprehension problems while digitally capturing studentsí responses in a manner that eliminates a labor-intensive scoring process. Ultimately, they will develop three formats of valid and reliable online reading comprehension assessments: 1) an Open ORCA; 2) a Closed ORCA; and 3) a Multiple Choice ORCA. They will also develop an administration and scoring document for each format to ensure standardization.

Research Design and Methods: The research team will develop the materials using an iterative process that will analyze and maximize reliability and validity of scores. They will evaluate how each format performs in relation to four relevant measures of contextual validity: 1) offline reading comprehension ability; 2) access to the Internet at home; 3) level of technology integration at school; and 4) the districtís economic status. After developing these materials, they will present all of the data to a diverse panel of educational leaders and conduct a practicality evaluation of each instrument.

Control Condition: Due to the nature of the research design, there is no control condition.

Key Measures: Key measures include scores from the drafts of the ORCA test forms, offline reading comprehension measures from the NAEP, and short online surveys to assess studentsí access to and use of the internet. The researchers will also use qualitative data from focus groups and district- and state-supplied information (e.g. demographic).

Data analytic strategy: The data from the iterative cycles, focus groups, and small and large scale pilot studies will be combined with a sequence of mathematical and modeling analyses, such as Item Response Theory, Multi Facet Rasch Modeling, and Structural Equation Modeling, to deliver the best set of online reading comprehension assessments.

Related IES Projects: Developing Internet Comprehension Strategies Among Adolescent Students At Risk to Become Dropouts (R305G050154)

Products and Publications

Book chapter

Leu, D.J., Everett-Cacopardo, H., Zawilinksi, L., McVerry, J.G., and O'Byrne, W.I. (2012). The New Literacies of Online Reading Comprehension. In C.A. Chapelle (Ed.), The Encyclopedia of Applied Linguistics (pp. 4239–4247). Oxford: Wiley-Blackwell.

Leu, D.J., Forzani, E., Burlingame, C., Kulikowich, J., Sedransk, N., Coiro, J., and Kennedy, C. (2013). The New Literacies of Online Research and Comprehension: Assessing and Preparing Students for the 21st Century With Common Core State Standards. In S.B. Neuman, L.B. Gambrell, and C. Massey (Eds.), Quality Reading Instruction in the Age of Common Core Standards (pp. 1–11). Newark, DE: International Reading Association.

Leu, D.J., Forzani, E., and Kennedy, C. (2014). Providing Classroom Leadership in New Literacies: Preparing Students for Their Future. In S.B. Wepner, D.S. Strickland, and D. Quatroche (Eds.), The Administration and Supervision of Reading Programs, 5th Edition (5th ed.). New Yorki: Teachers College Press.

Leu, D.J., Kinzer, C.K., Coiro, J., Castek, J., and Henry, L.S. (2013). New Literacies: A Dual Level Theory of the Changing Nature of Literacy, Instruction, and Assessment. In D.E. Alvermann, N.J. Unrau, and R.B. Rudell (Eds.), Theoretical Models and Processes of Reading (6th ed., pp. 1150–1181). Newark, NJ: International Reading Association.

Journal article, monograph, or newsletter

Coiro, J. (2012). The New Literacies of Online Reading Comprehension: Future Directions. The Educational Forum, 76(4): 412–417.

Kennedy, C., Rhoads, C., and Leu, D.J. (2016). Online Research and Learning in Science: A One-To-One Laptop Comparison in Two States Using Performance Based Assessments. Computers & Education, 100, 141–161.

Leu, D.J., and Forzani, E. (2013). New Literacies in a Web 2.0, 3.0, 4.0,.. [infinity] World. Research in the Schools, 19(1): 75–81.

Leu, D.J., Forzani, E., and Kennedy, C. (2015). Income Inequality and the Online Reading Gap: Teaching Our Way to Success With Online Research and Comprehension. The Reading Teacher, 68(6): 422–427.

Leu, D. J., Forzani, E., Rhoads, C., Maykel, C., Kennedy, C., and Timbrell, N. (2015). The New Literacies of Online Research and Comprehension: Rethinking the Reading Achievement Gap. Reading Research Quarterly, 50(1), 37–59.

Leu, D.J., McVerry, J.G., O'Byrne, W.I., Kiili, C., Zawilinski, L., Everett-Cacopardo, H., Kennedy, C., and Forzani, E. (2011). The New Literacies of Online Reading Comprehension: Expanding the Literacy and Learning Curriculum. Journal of Adolescent and Adult Literacy, 55(1): 5–14.

** This project was submitted to and funded under Interventions for Struggling Adolescent and Adult Readers and Writers in FY 2009.