Project Activities
The project will begin by developing learning maps for two math and two science topics. Based on these progressions, iterative cycles of ONPAR task development, cognitive labs, and reviews will occur and tasks will be placed into topic-specific modules. The modules will be investigated in experimental trials comparing the performance of ELLs and native English speakers on the new modules and traditional assessments designed to assess the same materials.
Structured Abstract
Setting
This study will take place in large, urban schools that are members of the Council of the Great City Schools across the U.S. Sample: The development studies will include 50 middle school ELLs and native-English speaking students. The experimental study will include approximately 1,200 middle-school ELLs who either speak Spanish or one other language to be determined prior to the study and their native speaking peers.
Assessment
The tasks for this study will be designed for delivery via computer. Key components of the proposed assessments include interactive, multipart performance tasks that dynamically present experimental or authentic contexts and target questions, using multiple screens and a range of representations (e.g., simulations, animations, image rollovers, sound, and interactive sequences) in addition to essential text and oral support as needed for precision. Depending on the nature of the task questions, students will be asked to respond by building, modeling, assembling, categorizing, or producing relational or inferential explanations using screen stimuli. Some learning loops will be tested within the modules where students will have the opportunity to revise their responses based on proposed learning progression criteria. A total of about 30 tasks, selected closed-response items, and four middle school end-of-unit modules with a total of approximately 35 score points each, will be constructed to measure terminal and supporting KSAs. After the modules are completed, students and teachers will be provided with immediate performance data, guidance, and results of how students responded to intramodule learning.
Research design and methods
One-to-one cognitive lab interviews with approximately 50 students will investigate how well students understand the tasks and items they are given; the accessibility of the overall approach and the response types; and the usefulness of the intramodule feedback and types of information students will receive during testing. Findings will be used to refine the tasks and associated materials. The experimental trials will compare performance of the three groups on prototype item modules and traditional item modules measuring the same or similar content, while controlling for science or mathematics ability using an independent measure. Additionally, at the module and task levels, relationships among ELP, native language, L1 proficiency, and school variables for the 3 groups and 2 forms will be inspected. Content specialists will review the tasks for potential bias and access problems.
Key measures
Besides the modules, the measures will include researcher-developed teacher rating of student math skills, cognitive labs and interview protocols, educator questionnaires, STAR Math assessment (Renaissance Learning), and state test data in math, science, reading, and English language proficiency.
Data analytic strategy
Student performances between ONPAR and traditional tasks and forms and among groups will be compared. Structural relationships between forms, school and classroom makeup, and groups will be conducted at the module levels. If possible, factor structures for each form by group will be analyzed. Qualitative investigation of the cognitive lab findings will also focus on effective and not effective ONPAR task features for ELLs and native English speakers.
People and institutions involved
IES program contact(s)
Products and publications
Products: Proof-of-concept formative tasks and four end-of-unit modules for middle school ELLs will be produced. Detailed feedback and performance reports for students and teachers will be developed to characterize learning along the learning progression pathways. A report and peer reviewed publications will also be produced.
ERIC Citations: Find available citations in ERIC for this award here.
Supplemental information
Task development studies will attempt to control for ELL math or science knowledge in order to evaluate performance on the assessments for students with similar skills levels and language proficiency levels. A mini-study will be conducted to investigate how accurately teachers evaluate the math knowledge of ELLs in their classrooms. Feasibility and utility data related to the modules and associated materials for teachers, students, and prospective decision makers will be gathered from educators during development, testing, and post-trials using questionnaires and interview protocols. Data will also be gathered from students during the lab interviews. These results will be coded and qualitative content analysis procedures will be used to examine interest, value, and ease for use in classrooms.
An experimental study of the new ONPAR tasks and modules will be conducted in the third year of the grant. About 1,200 ELLs and native speakers will be randomly assigned within stratas (defined by geographic region, SES, school make-up, type of English language acquisition program, time in country, and home language) to take either the ONPAR modules or a traditional module specifically designed by an independent contractor to measure the same content and cognitive complexity. Control Conditions: In the control condition, students will receive traditional school assessment forms of the same content.
Questions about this project?
To answer additional questions about this project or provide feedback, please contact the program officer.
