Effect of Linguistic Modification of Math Assessment Items on English Language Learner Students
Intervention description
This study investigates how linguistic modification affects students' ability to access and respond to mathematics content on standardized achievement tests. Linguistic modification purposefully alters the language of test items, directions, and/or response options—by reducing sentence length and complexity, using common or familiar words, and inserting concrete language (Abedi 2008; Abedi, Lord, and Plummer 1997; Sato 2008; Sireci, Li, and Scarpati 2002)—to clarify and simplify the text without simplifying or significantly altering the construct (concepts, knowledge, skills) tested. Such modification should reduce the language load of the text in a mathematics item.
Six principles guide linguistic modification:
- Using research-based strategies, eliminate construct-irrelevant language (unessential language and unfamiliar culture-specific references, for example) from the item without altering the mathematics constructs being assessed.
- Using research-based strategies, reduce the language load without altering the construct being assessed.
- Using expert judgment, remain faithful to unmodified item's characteristics—content and target construct assessed, item type and cognitive complexity.
- Linguistically modify items in intentional, defensible, and focused ways that address the specific linguistic needs of English language learner students—do not use a one size fits all strategy.
- Document all linguistic modifications and associated rationales.
- Monitor the impact of linguistic modification through ongoing review and feedback (qualitative and quantitative).
In the initial stage of the study, experts with specialties in mathematics, applied linguistics, measurement, curriculum and instruction, English language development, and the English language learner population developed two sets of items to measure the effects of linguistic modification on student access to test content. The work group first collected released test items from the National Assessment of Educational Progress and the California Standardized Testing and Reporting (STAR) program web sites. Measurement and content specialists at the national and state levels extensively reviewed the items and found them to be psychometrically sound, aligned to state content standards, and developmentally appropriate for students in grades 7 and 8. (The pooled items represent the broad content strands of all states' standards in mathematics.)
The work group determined that the items most amenable to research-based linguistic modification strategies came from two mathematics content strands: measurement, and numbers and operations. From these two strands, they selected the items most appropriate for linguistic modification and applied the linguistic modification strategies, producing two mathematics item sets—one with linguistically modified items and one with unmodified items.
The treatment intervention in this study was the item set containing 25 linguistically modified mathematics items. The counterfactual was the item set with the original unmodified items. Initial study activities focused on ensuring that the two item sets were sufficiently valid for large-scale data collection efforts. This included conducting cognitive interviews (think-aloud protocols) with English language learner and non–English language learner students in grades 7 and 8 and pilot testing linguistically modified items with a sample of 100 such students.