Project Activities
The research team developed assessments of vocabulary depth that focus on the measurement of vocabulary learning in content area classrooms. The project team developed a model of vocabulary breadth, as well as methods to measure vocabulary knowledge systematically at different levels of depth. The team used the vocabulary breadth model and the vocabulary depth measures together to create profiles of middle grade student vocabulary knowledge over at least two specific content areas (e.g., world history and middle school biology) within the larger domains of history, geography and biology.
Structured Abstract
Setting
The study population was comprised of students in Grades 7 and 8 from across the United States from six urban, six suburban, and eight rural schools from Alabama, Arkansas, Arizona, California, Connecticut, Georgia, Iowa, Idaho, Illinois, Indiana, Kentucky, Nevada, and Tennessee.
Sample
A total of 1,449 seventh grade and 1,622 eighth grade students participated in this study.
Three types of items were developed for and tested in this study—each with specific goals in mind to test different types, and hopefully depth, of vocabulary knowledge. These types included an idiomatic associates item type, intended to measure familiarity with patterns of usage; a topical associates item type, intended to measure the kinds of associations represented in semantic memory; and the hypernym item type, designed to measure access to conceptual representations and associated patterns of inference.
Research design and methods
The goal of the study was to explore whether the use of the three different types of multiple-choice items could reveal distinguishable levels of depth of vocabulary understanding from middle school students. A within-subjects design was incorporated in order to measure students’ performances on all three item types for the same set of academic vocabulary words. Items were created for two sets of 10 vocabulary words to which students were exposed to one set of 10 or the other; hence, each student was exposed to each word within his or her assigned set three times in the context of the three item types.
Key measures
Items were created for two sets of 10 vocabulary words to which students were exposed to one set of 10 or the other; hence, each student was exposed to each word within his or her assigned set three times in the context of the three item types. The instrument also included an anchor set of 20 items drawn from SERP's Word Generation project.
Data analytic strategy
The researchers performed initial item analysis, item response theory analyses, and validity analyses.
Key outcomes
- This project developed a new approach to vocabulary assessment that measures different aspections of vocuablary knowledge, using natural language processing to support item design (Deane et al 2014).
- The team developed vocabulary topic maps that provide detailed word lists to guide instruction and assessment in the content areas of history, geography, and biology and valid item-battery to assess partial vocabulary knowledge.
- Several studies explored the use of multiple item types for the same word to support richer inferences about students’ states of partial vocabulary knowledge using domain-specific vocabulary.
People and institutions involved
IES program contact(s)
Products and publications
The outcomes of this research include vocabulary topic maps that provide detailed word lists to guide instruction and assessment in the content areas of history, geography, and biology, a valid item-battery to assess partial vocabulary knowledge, a set of assessments to measure breadth and depth of vocabulary knowledge in two specific domains (world history and biological science) at the middle school level, and published reports describing these outcomes.
Publications:
Book chapter
Deane, P. (2012). NLP Methods for Supporting Vocabulary Analysis. In J.P. Sabatini, T. O'Reilly, and E. Albro (Eds.), Reaching an Understanding: Innovations in how we View Reading Assessment (pp. 117-144). Lanham, MD: Rowman and Littlefield.
Nongovernment report, issue brief, or practice guide
Deane, P., Lawless, R.R., Li, C., Sabatini, J., Bejar, I.I. and O'Reilly, T. (2014). Creating Vocabulary Item Types That Measure Students' Depth of Semantic Knowledge. Washington, DC: ETS.
Proceeding
Krovetz, R., Deane, P., and Madnani, N. (2011). The Web is not a Person, Berners-Lee is not an Organization, and African-Americans are not Locations: An Anlaysis of the Performance of Named-Entity Recognition. In Proceedings of the ACL 2011 Workshop on Multiword Expressions: From Parsing and Generation to the Real World (MWE 2011) (pp. 57-64). Portland, OR: Association for Computational Linguistics.
Related projects
Questions about this project?
To answer additional questions about this project or provide feedback, please contact the program officer.
