Co-Principal Investigators: Paulina Kulesz (University of Houston), Joshua Lawrence (University of Oslo)

Purpose: The researchers seek to improve understanding of factors that affect assessment of vocabulary knowledge among English learners (ELs) in unintended ways by conducting multiple item-level analyses. Based on those findings, the researchers will refine an existing academic vocabulary assessment to be psychometrically and theoretically sound for these students. Results will expand what is known about item characteristics that are differentially difficult for ELs, and will provide a set of test recommendations for improving the accuracy and reliability of previously developed vocabulary assessments.

Project Activities: The first phase of the study involves secondary data analysis on item-level data that has collected from a prior IES-funded randomized efficacy trial of Word Generation. The researchers will test whether English proficient language minority students (many of whom formerly used to be ELs) and ELs respond significantly differently to academic vocabulary test items. In the second phase of the study, the researchers will refine and test our new items with data collected from a new sample of students. The analysis will involve Explanatory Item Response Models, Differential Item Functioning, and Differential Distractor Functioning.

Products: Researchers will produce valid and reliable assessment materials which will be available on a website developed specifically for the research project and link to the relevant resources, research instruments, and contextualizing information. Researchers will also produce peer-reviewed publications.

Structured Abstract

Setting: This study will take place in Texas.

Sample: Roughly 13,780 middle school students, who participated in the earlier Word Generation efficacy trial, contributed to the existing data set. Approximately 22% of the participants were English-only students, 12% were initially fluent ELs (started 1st grade as proficient), 47% were ELs who were redesignated as proficient, and 18% were limited English proficiency students. Researchers will conduct item-level analyses with data collected from these students. For the second phase of the study, researchers plan to recruit approximately 1,100 students in Grades 6–8.

Assessment: Researchers will use the Word Generation Academic Vocabulary Assessment developed in the previous IES grant. The second phase of the study will involve an updated version of the assessment.

Research Design and Methods: Researchers use explanatory item response models to examine differential effects of target word characteristics on responses to the Word Generation Academic Vocabulary Test for English-only and ELL students. They also conduct differential item functioning (DIF) and differential distractor functioning (DDF) to examine which test items are biased for ELLs. Test items will be revised based on these results. In the third year, the final measure will be administered to more than 1,000 students in three schools to examine whether the revised assessment more reliably measures ELL students' academic vocabulary knowledge.

Control Condition: The assessment will compare results with English-Only students.

Key Measures: Researchers will code existing data from the Word Generation Academic Vocabulary Test to understand what factors make an item hard of ELLs in unintended ways. They will examine the target words' part of speech, frequency, specificity, polysemy, dispersion, word length, and semantic similarity to the key. They will also examine other item features like semantic complexity and the semantic relationship between keys, distractors and target words. They will refine the test based on the results of these analyses and determine whether the refined items are more reliable for ELLs.

Data Analytic Strategy: The research team will conduct explanatory item response theory, DIF and DDF analyses to examine bias in the Word Generation Academic Vocabulary Test for assessing ELL students' vocabulary knowledge. The analysis on the revised assessment will utilize the same analytic strategy as the first phase.

Related IES Projects: Word Generation: An Efficacy Trial (R305A090555)