Effect of Linguistic Modification of Math Assessment Items on English Language Learner Students
Study design
Using a 2-by-3 fully crossed design, the study tests middle school students in grades 7 and 8 in the spring and summer of 2008 with linguistically modified items and unmodified items. The factors are item sets (unmodified and linguistically modified) and student population; grade level (7 or 8) serves as a blocking factor. The student population for each item set consists of 600 English language learner students, 600 non–English language learner students at low reading levels, and 600 non–English language learner students at high reading levels, equally divided between grade 7 and 8 students.
Students are randomly assigned to a treatment or control condition within each school, grade, and English proficiency group. Box 1 summarizes the study design.
The sample requirements were determined through analyses of the power of the statistical tests used, which include analysis of variance and confirmatory factor analysis. With the intended cell sizes of 3,600 students, this design provides a minimum detectable effect size of 0.20 standard deviation (power = 0.80 and alpha = 0.05) for the main research question addressed by the analysis of variance, which asks whether the score difference between the unmodified and linguistically modified item sets differs for the English language learner and non–English language learner student subgroups. Because researchers anticipated challenges in matching achievement history data to each student participant, the study over sampled; thus the actual sample size is around 4,600 students to ensure sufficient sample sizes in each cell following the matching process.
Box 1. Study features
| Study design | • Experimental design. |
|---|---|
| Implementation | • Spring/summer 2008–fall 2008. |
| Treatment | • Item set (treatment group is given set with the linguistically modified items; control group is given set with unmodified items). |
| Unit of assignment | • Random assignment at the student level. |
| • Each student randomly assigned to a condition within each school and grade. | |
| • Each student randomly assigned to the control (administered an item set with the unmodified items) or treatment condition (administered an item set with the linguistically modified items). | |
| • Achievement history data collected after testing for each student and matched through student identifiers. Data collected included state standardized test scores in English language arts and mathematics and, for English language learner students only, English-language proficiency scores (California English Language Development Test). | |
| Student recruitment | • Participating schools submitted rosters of all students in grades 7 and 8 who met eligibility criteria for the state assessment; in most cases this included all students in intact math classrooms. |
| Sample characteristics | • 4,600 students enrolled in grades 7 or 8 in public schools in California.1 |
| • The target English language learner student sample consisted of students whose first language is Spanish who have early-intermediate to high levels of English language proficiency.2 | |
| • The non–English learner sample consisted of general education students who are English proficient. These students were divided into two groups based on state achievement test scores in English language arts: those with high reading ability and those with low reading ability. | |
| Informed consent | • Letters informing parents and guardians about the study were sent home with eligible students. A passive parent consent process was approved for this study |
| 1 To control for cross-state differences in the math content standards of state assessments, students were sampled from only one state. | |
| 2 By studying only native speakers of Spanish, the study controls for sources of variability related to native language. Spanish was selected as the language for the study because 75 percent of English language learner students in the West Region (California, Nevada, Utah, and Arizona) identify Spanish as their primary or secondary language. We limit the population to students who have sufficient levels of proficiency in the English language to benefit from linguistic modification of test items, as students who cannot yet read English are less likely to benefit from this accommodation. | |