Purpose:  The lack of research-based evidence on how best to provide appropriate testing accommodations for English language learners (ELLs) creates challenges for policymakers and practitioners in ensuring that results provide valid results for ELL students. This project will examine the effectiveness and validity of the language-based accommodations that are most commonly used for Spanish-speaking ELL students. Studies will focus on the validity of assessments under the different accommodations and whether there are differential impacts of the accommodations for students who differ on such variables as language proficiency (both in their first language and in English), length of time in the United States, and instructional program.

Project Activities: The proposed study will be conducted in three phases. In Phase I, a computer-based assessment and accommodation system will be developed and field-tested. In Phase II, the system will be utilized to examine the effectiveness and validity of five commonly used language-based accommodations. In Phase III, the system will deliver accommodations that were shown in Phase II to be effective and valid in selecting accommodations that are tailored to the individual needs of each ELL.

Products: The products of this project will be a fully developed and validated  computer-based assessment system that measures middle-school math and provides accessible and appropriate accommodations for ELL students. This system will include a short language proficiency assessment in English and in Spanish. Peer-reviewed publications will also be produced.

Structured Abstract

Setting:   This study will be conducted in middle schools in three states (California, New Jersey, New Mexico) with large numbers of Spanish-speaking ELL students.

Population:  The population for this study is a total of 3,600 Spanish-speaking and native English-speaking students in eighth grade.

Assessment:  This project will create a computer-based math assessment and accommodation system for students in eighth grade designed to help gather better evidence of math knowledge for English learners. The assessment will be designed to align with both the Common Core State Standards in Math and reviewed for alignment with the standards for each state in the study.  The five accommodations include:  (1) linguistically modified version of the test, (2) bilingual versions of the test (native language assessment), (3) English glossary, (4) bilingual glossary, and (5) read-aloud test items.

Research Design and Methods:  Phase I of the study involves system development and alpha and beta testing for a new math assessment and five associated accommodations for ELLs.  Researchers will create the required item pool by first identifying available items from the participating states’ released tests. They will determine which of these items are aligned with the Common Core State Standards (CCSS) for Grade 8 Mathematics. Next, they will create additional items as needed to ensure that the final item pool provides coverage of all of the topics for this subject and grade level as specified in the CCSS. Next, items for the five accommodations will be created to align with the assessment.

Phase II includes a randomized control trial study in which ELL and non-ELL students in eighth grade will be assigned to one five different accommodation conditions or a control condition in which no accommodations are provided. Background data on student characteristics, particularly proficiency in English and Spanish and language of instruction, will be collected to inform Phase III of the study. Results will be used to complete the computer-based assessment and accommodation system. A small-scale, concurrent and retrospective verbal protocol analysis (10 ELLs and 10 native English speakers per each accommodation) will be conducted to provide additional insight into the effects and validity of the given accommodations.

Phase III will investigate the effectiveness of this new system for selecting appropriate accommodations for each ELL. For example, if a student is proficient in both Spanish and English and is instructed in English and still is at the ELL status (needing linguistic support), then such English-language accommodations as providing a linguistically modified version and/or English or bilingual glossary may be recommended. If a student is proficient in both English and Spanish but instructed in Spanish then a bilingual version of the assessments and/or bilingual glossary may be recommended.

Control Condition:  In Phase II, ELLs will be randomly assigned to take the assessment with or without accommodations. Non-ELLs will be randomly assigned to take the test without accommodations, or with the one of the accommodations that does not require knowledge of Spanish.

Key Measures: The key outcome measure will be scores on the math assessment developed in Phase I of the grant. Scores from state-based tests of English Language Proficiency will also be collected and the Timed Index to Measure English Reading (TIMER) will be administered in both English and Spanish.

Data Analytic Strategy:  Performance of ELL and non-ELL students in mathematics will be compared using a three-level hierarchical linear model (students within classrooms and classrooms within schools) in which performance on the math test is predicted by the accommodation received and student  covariates. A criterion-related validation approach will be conducted through multiple group  confirmatory factor analyses in which a set of hypothesis regarding invariance between accommodated and non-accommodated students will be tested.