Project Activities
The researchers developed and validated a set of 27 assessment tasks that were designed to measure students' three-dimensional understanding of energy. To support the use of the tasks, support materials were developed, including alignment to NGSS, task and item level descriptions of what students are expected to do when responding, scoring guidance, and sample student responses. Professional development materials were also developed to increase teachers’ understanding of how to design and use NGSS-aligned assessments.
Structured Abstract
Setting
The setting for this study included elementary, middle, and high schools from urban, rural, and suburban locations across the U.S.
Sample
The final sample included 83 teachers and 3,446 students in grades 4 through 12 across a range of demographics. Approximately 14% of the sample were elementary school students, 45% were middle school students, and 40% were high school students. Approximately 7% of the students indicated that English was not their primary language. There were about equal numbers of male and female students. A little over half of the students identified themselves as White, 14% identified as Hispanic, 9% identified as Black, and 6% identified as Asian.
The researchers developed and validated a set of 27 three-dimensional assessment tasks focused on energy that were aligned to elementary, middle, and high school level Next Generation Science Standards (NGSS). The target NGSS disciplinary core ideas included Definitions of Energy, Conservation of Energy and Energy Transfer, Relationship Between Energy and Forces, Energy in Chemical Processes and Everyday Life, and Organization for Matter and Energy Flow in Organisms. The targeted NGSS science practices include Developing and Using Models, Planning Investigations, Analyzing and Interpreting Data, Constructing Explanations, and Engaging in Argument from Evidence. The target NGSS crosscutting concepts included Patterns; Cause and Effect; and Energy and Matter. The set of 27 tasks included 7 elementary school tasks, 10 middle school tasks, and 10 high school tasks. Each task presents a real-world phenomenon followed by a series of constructed-response and multiple-choice items that scaffold students in making sense of the phenomenon. To increase the practical utility of the assessment tasks, the researchers developed supporting materials to guide the use of the tasks and the interpretation of the results. They also created professional development materials to support assessment developers and classroom teachers in creating their own three-dimensional assessments.
Research design and methods
To develop the assessment, researchers began by selecting NGSS performance expectations (PEs) related to elementary, middle, and high school energy ideas. The PEs were used to define the target practices, crosscutting concepts, and core ideas. Then, three-dimensional assessment tasks were developed. The tasks are built around real-world phenomena and are made up of between 3 and 11 multiple-choice and constructed-response items. Each item within the task requires students to use multiple dimensions (e.g. the modeling practice and the concept of conservation of energy) to make sense of something or figure something out. The last item in most tasks requires students to pull together the pieces they discovered along the way to develop an explanation of the phenomenon. Each multi-item task was intended to provide a picture of students’ three-dimensional understanding of some aspect of the energy concept. Additionally, some of the items were written using both a multiple choice (MC) and constructed response (CR) format to help instrument developers select sets of items that balance the time and effort that students need during testing and scorers need to evaluate responses.
The tasks underwent several rounds of review and revision during the development process, including pilot testing and think-aloud interviews with students in grades 4 through 12 and two cycles of expert review. During pilot testing, students from across the U.S. were asked to respond to one task and a set of feedback questions. Those questions allowed students to provide feedback about the comprehensibility of the task, any difficulties they had with the test features (i.e. the drawing tool), and how interesting and relevant the tasks were to them. During think-aloud interviews, a member of the research team observed a student verbalizing their thoughts as they responded to a task. The interviews provided information about the readability of the tasks, the cognitive processes that the students used while responding, and the extent to which the students found the tasks engaging. The expert reviews were conducted by advisory board members with science education and content knowledge expertise and experience in implementing and/or assessing NGSS. The reviewers evaluated the scientific accuracy of the assessments, the alignment of the assessments to the NGSS, the fairness and comprehensibility of the assessments, and the grade appropriateness and usability of the scoring rubrics. In the final step of development, the tasks were field tested with a larger student population to gather data that was used to evaluate the reliability and validity of the tasks and answer the research questions. During the field test, students responded to three tasks and a set of 14 stand-alone, MC linking items.
Control condition
Not applicable for this study.
Key measures
The key measures for this study include students' written responses to constructed response items, students' answer choice selections to multiple choice items, students' responses during interviews, reviewer feedback on draft items, and teachers' feedback on the supporting materials.
Data analytic strategy
The project included qualitative analyses of students' written comments and interview responses and Rasch modeling of pilot and field test data. Researchers used Wright maps to ensure that the assessment tasks were targeting the ability levels of the different grade bands. The research team also used a principal component analysis of the Rasch residuals to investigate whether the items within the tasks were measuring the same construct.
People and institutions involved
IES program contact(s)
Project contributors
Products and publications
Project website:
Publications:
Find available citations in ERIC for this award here.
Additional project information
Previous award details:
Related projects
Questions about this project?
To answer additional questions about this project or provide feedback, please contact the program officer.
