Purpose: English Language Learners, on average, score lower on math and science assessments than their English-speaking peers. Growing evidence suggests that ELLs who have the opportunity to develop English proficiency and adequate content coverage, including coverage of more complex concepts and skills, may over time do as well as, or better than, their native English-speaking peers. Given that many ELLs are still mastering English, the challenge is to give them access to more complex subject matter throughout their schooling even though they may not yet have the language to learn this material using traditional methods. The purpose of this grant is to develop proof-of-concept, classroom-embedded assessments that use multiple semiotic methods, including language, to reduce the overall English language load typically associated with the more difficult knowledge, skills, and abilities in middle school math and science. These assessments are intended to support and encourage teachers to teach more complex math and science to ELLs. The project will use an existing methodology called ONPAR (Obtaining Necessary Parity Through Academic Rigor) that has already been used effectively to develop summative assessment tasks (see http://iiassessment.wceruw.org/ for examples of this approach). Using auto-scoring techniques, this project will extend the use of ONPAR to create formative assessments capable of providing immediate feedback to students and teachers, and producing specific, individualized, data-driven guidance targeted to improving instruction for ELLs.

Project Activities: The project will begin by developing learning maps for two math and two science topics. Based on these progressions, iterative cycles of ONPAR task development, cognitive labs, and reviews will occur and tasks will be placed into topic-specific modules. The modules will be investigated in experimental trials comparing the performance of ELLs and native English speakers on the new modules and traditional assessments designed to assess the same materials.

Products: Proof-of-concept formative tasks and four end-of-unit modules for middle school ELLs will be produced. Detailed feedback and performance reports for students and teachers will be developed to characterize learning along the learning progression pathways. A report and peer reviewed publications will also be produced.

Structured Abstract

Setting: This study will take place in large, urban schools that are members of the Council of the Great City Schools across the U.S. Sample: The development studies will include 50 middle school ELLs and native-English speaking students. The experimental study will include approximately 1,200 middle-school ELLs who either speak Spanish or one other language to be determined prior to the study and their native speaking peers.

Assessment: The tasks for this study will be designed for delivery via computer. Key components of the proposed assessments include interactive, multipart performance tasks that dynamically present experimental or authentic contexts and target questions, using multiple screens and a range of representations (e.g., simulations, animations, image rollovers, sound, and interactive sequences) in addition to essential text and oral support as needed for precision. Depending on the nature of the task questions, students will be asked to respond by building, modeling, assembling, categorizing, or producing relational or inferential explanations using screen stimuli. Some learning loops will be tested within the modules where students will have the opportunity to revise their responses based on proposed learning progression criteria. A total of about 30 tasks, selected closed-response items, and four middle school end-of-unit modules with a total of approximately 35 score points each, will be constructed to measure terminal and supporting KSAs. After the modules are completed, students and teachers will be provided with immediate performance data, guidance, and results of how students responded to intramodule learning.

Research Design and Methods: One-to-one cognitive lab interviews with approximately 50 students will investigate how well students understand the tasks and items they are given; the accessibility of the overall approach and the response types; and the usefulness of the intramodule feedback and types of information students will receive during testing. Findings will be used to refine the tasks and associated materials. The experimental trials will compare performance of the three groups on prototype item modules and traditional item modules measuring the same or similar content, while controlling for science or mathematics ability using an independent measure. Additionally, at the module and task levels, relationships among ELP, native language, L1 proficiency, and school variables for the 3 groups and 2 forms will be inspected. Content specialists will review the tasks for potential bias and access problems.

Task development studies will attempt to control for ELL math or science knowledge in order to evaluate performance on the assessments for students with similar skills levels and language proficiency levels. A mini-study will be conducted to investigate how accurately teachers evaluate the math knowledge of ELLs in their classrooms. Feasibility and utility data related to the modules and associated materials for teachers, students, and prospective decision makers will be gathered from educators during development, testing, and post-trials using questionnaires and interview protocols. Data will also be gathered from students during the lab interviews. These results will be coded and qualitative content analysis procedures will be used to examine interest, value, and ease for use in classrooms.

An experimental study of the new ONPAR tasks and modules will be conducted in the third year of the grant. About 1,200 ELLs and native speakers will be randomly assigned within stratas (defined by geographic region, SES, school make-up, type of English language acquisition program, time in country, and home language) to take either the ONPAR modules or a traditional module specifically designed by an independent contractor to measure the same content and cognitive complexity. Control Conditions: In the control condition, students will receive traditional school assessment forms of the same content.

Key Measures: Besides the modules, the measures will include researcher-developed teacher rating of student math skills, cognitive labs and interview protocols, educator questionnaires, STAR Math assessment (Renaissance Learning), and state test data in math, science, reading, and English language proficiency.

Data Analytic Strategy: Student performances between ONPAR and traditional tasks and forms and among groups will be compared. Structural relationships between forms, school and classroom makeup, and groups will be conducted at the module levels. If possible, factor structures for each form by group will be analyzed. Qualitative investigation of the cognitive lab findings will also focus on effective and not effective ONPAR task features for ELLs and native English speakers.