Purpose: Vocabulary development is a critical part of learning to read well, and appears to be a significant aspect of the gap between competent and struggling readers. Teaching vocabulary directly can help enhance vocabulary learning and reading comprehension. While the importance of vocabulary development may be apparent to researchers and practitioners, the state of the art in vocabulary assessment is still rather limited. The purpose of this project is to develop improved methods for measuring vocabulary and word learning in specific subject areas such as social studies or science.

Project Activities: The research team will develop assessments of vocabulary depth that focus on the measurement of vocabulary learning in content area classrooms. The project team will develop a model of vocabulary breadth, as well as methods to measure vocabulary knowledge systematically at different levels of depth. The team will then use the vocabulary breadth model and the vocabulary depth measures together to create profiles of middle grade student vocabulary knowledge over at least two specific content areas (e.g., world history and middle school biology) within the larger domains of history, geography and biology.

Products: The expected outcomes of this research include vocabulary topic maps that provide detailed word lists to guide instruction and assessment in the content areas of history, geography, and biology, a valid item-battery to assess partial vocabulary knowledge, a set of assessments to measure breadth and depth of vocabulary knowledge in two specific domains (world history and biological science) at the middle school level, and published reports describing these outcomes.

Structured Abstract

Setting: Participating schools are located in a large urban city in Massachusetts.

Population: Participants include world history and biology students in at least four middle schools, with approximately 120 students per school per grade. Sampling of schools and students will be done such that student diversity in terms of race/ethnicity, socioeconomic status, and English language learner status are well represented.

Research Design and Methods: The research will fall into three strands. Strand I will focus on modeling the fine topical structure of vocabulary, and on using such a model to define measures of topic-specific vocabulary breadth. Strand II will focus on creating appropriate measures of vocabulary depth. Strand III will bring Strands I and II together in a large-scale study of a middle school student population. In this large field study, assessments of content-area vocabulary that tap both breadth and depth of knowledge will be created using both convergent and discriminant validity.

Key Measures: Key measures include researcher-constructed tests with four different forms that test the same item types, the Peabody Picture Vocabulary Test-Revised, and the SARA reading battery. Researchers will also collect demographic data such as native language, course grades, and score on the last state standardized accountability measure.

Data Analytic Strategy: The researchers will calculate mean accuracy scores for each item type. They expect mean accuracy to vary widely across item types, and to fall into a natural order, reflecting whether the item type is measuring low or high measures of vocabulary depth. An item analysis will also be performed. Finally, the researchers will examine correlations between test scores and external vocabulary and content knowledge measures.

Products and Publications

Book chapter

Deane, P. (2012). NLP Methods for Supporting Vocabulary Analysis. In J.P. Sabatini, T. O'Reilly, and E. Albro (Eds.), Reaching an Understanding: Innovations in how we View Reading Assessment (pp. 117–144). Lanham, MD: Rowman and Littlefield.

Nongovernment report, issue brief, or practice guide

Deane, P., Lawless, R.R., Li, C., Sabatini, J., Bejar, I.I. and O'Reilly, T. (2014). Creating Vocabulary Item Types That Measure Students' Depth of Semantic Knowledge. Washington, DC: ETS.

Proceeding

Krovetz, R., Deane, P., and Madnani, N. (2011). The Web is not a Person, Berners-Lee is not an Organization, and African-Americans are not Locations: An Anlaysis of the Performance of Named-Entity Recognition. In Proceedings of the ACL 2011 Workshop on Multiword Expressions: From Parsing and Generation to the Real World (MWE 2011) (pp. 57–64). Portland, OR: Association for Computational Linguistics.