Effect of Linguistic Modification of Math Assessment Items on English Language Learner Students
Data collection approach
Data collection involved six sources of data: expert review, cognitive interviews, item tryouts, operational administration of the item sets, student language background survey, and student achievement history data.
Expert review. Experts in mathematics, the English language learner student population, English language development, measurement, curriculum and instruction, and applied linguistics together reviewed released items from state tests and National Assessment of Educational Progress and noted their appropriateness for study purposes. The experts linguistically modified test items, using research-based strategies.
Cognitive interviews. Using a think-aloud protocol, researchers elicited feedback from nine students in grades 7 and 8 about the strategies they use to solve mathematics problems.
Item tryouts. The two item sets were administered to 100 middle school English language learner and non–English language learner students.
Operational administration of item sets. Middle school students were randomly assigned to the treatment (item set with the linguistically modified items) or control group (item set with the unmodified items). Each item set included 25 multiple choice items.
Student language background survey. A five-item language background survey at the end of the item set booklet was used to provide additional information about the Spanish speakers in the English language learner analytic sample.
Student achievement history data. For all participating (tested) students, districts submitted archived achievement history data—state test scores in English language arts and mathematics and, for English language learner students, English language proficiency scores and proficiency level—from school records or the district database. Researchers from Regional Educational Laboratory West then matched archived data to student responses on the study's mathematics item set (linguistically modified or unmodified).