Project Activities
Researchers conducted a series of quantitative and qualitative studies to develop and test a theoretical framework to explain the relationships between English reading, English writing, and STE demands of open-response items for fifth grade ELs and non-ELs. Using data collected from the Massachusetts Comprehensive Assessment System (MCAS), researchers also explored if and how student English language proficiency and performance on multiple-choice STE items is related to performance on open-response items. Researchers on this project previously conducted a related study with funding from NCER to investigate sources of difficulty for ELs on multiple-choice items on STE assessments.
Structured Abstract
Setting
The project took place in 12 elementary schools in 3 urban school districts in Massachusetts.
Sample
Quantitative studies included responses for approximately 71,000 non-ELLs and 4,000 ELLs per year for 13 years (2003-2015) on the fifth grade MCAS STE assessments and responses for approximately 4,000 ELs per year on the English language proficiency assessments given in fifth grade. Qualitative studies included 122 ELLs and 17 non-ELLs selected to reflect the diverse ethnic, linguistic, and socioeconomic communities in urban districts in Massachusetts.
Intervention
This project examined the interactions between characteristics of open-response test items and characteristics of students. A theoretical framework was developed to describe how these characteristics interact, in order to inform better assessment and accommodation practices for ELs.
Research design and methods
Researchers conducted a series of four linked studies. Study 1 investigated how student test scores on the open-response test items on the STE MCAS can be predicted, using students' scores on the multiple-choice items on the STE MCAS, alongside EL and non-EL students' test scores on English Language Arts and English Language proficiency tests. Based on these findings, Study 2 developed: a) demand hypotheses about the relationships between the differences in performance of ELs and non-ELs on open-response STE test items and the English Reading, English Writing, and STE Demands of these items, and b) rubrics for coding the English Reading, English Writing, and STE Demands of open-response items. Study 3 tested these demand hypotheses by: a) conducting bilingual cognitive labs to elicit students' thinking as they responded to open-response test items; and b) analyzing quantitative data to identify items that contributed to differential performance for EL vs. non-EL students with similar STE proficiency levels. In Study 4, researchers explored how student-level characteristics such as English language proficiency level, and item-level characteristics such as English Writing, English Reading, and STE Demands interacted to contribute to differential performance of ELs and non-ELs on open-response items.
Control condition
Due to the nature of the research design, there was no control condition.
Key measures
Student demographic data; MCAS assessments in English language arts and STE; English language proficiency assessments.
Data analytic strategy
Quantitative studies used multiple regression, differential item functioning analysis, correlation analysis, and multinomial logistic regression. Qualitative studies used thematic and discourse analysis, interaction analysis, and grounded theory to identify and describe patterns across items and students.
Key outcomes
Several lines of investigation revealed preliminary evidence that ELs' status as English learners can negatively affect their ability to show what they know on STE open-response items, including findings that 1) both STE knowledge and skills and English language proficiency are important predictors of student performance on STE open-response items; 2) open-response items that require higher levels of English language writing proficiency particularly disadvantage ELs as compared to non-ELs.; and 3) an EL student is much more likely to omit a response to an open-response test item if (s)he has a lower level of English language proficiency, even when STE proficiency is controlled for. Findings of all studies are reported in Noble, Wells, & Rosebery, 2023.
People and institutions involved
IES program contact(s)
Products and publications
Publications:
Noble, T., Wells, C. S., & Rosebery, A. S. (2023). English Learners and Constructed-Response Science Test Items Challenges and Opportunities. Educational Assessment, 28(4), 246-272.
Related projects
Supplemental information
Co-Principal Investigator: Rosebery, Ann
Results can be used to inform better assessment and accommodation practices for ELs. Findings include: 1) an analysis of the words on STE open response test items that teachers identified as unfamiliar to their 5th grade EL students and 5th grade EL interviewees identified as confusing; 2) a synthesis of research on ELs' acquisition, development, and use of English language; 3) an analysis of Differential Item Functioning for open response STE items on the fifth grade MCAS; and 4) a manuscript describing the quantitative analyses that led us to conclude that English language proficiency affects ELs' scores on open-response test items, independent of their STE proficiency levels.
Questions about this project?
To answer additional questions about this project or provide feedback, please contact the program officer.
