Assessing Students' Progress on the Energy Concept Using Three-Dimensional Items (ASPECt-3D)
Previous Award Number: R305A170282
Previous Institution: American Association for the Advancement of Science (AAAS)
Co-Principal Investigator: George DeBoer
Purpose: The purpose of this project is to develop and validate assessments that can measure students' ability to use the three dimensions outlined by the Next Generation Science Standards—science and engineering practices, crosscutting concepts, and disciplinary core ideas—to make sense of energy-related phenomena. The project builds on a previously funded IES project in which the researchers developed three vertically-equated, multiple-choice instruments to assess students' progress on the energy concept. Those instruments focused mainly on assessing students' understanding of science content. The current project would expand on this work by developing item clusters that assess students' growth of understanding along the NGSS practice and crosscutting concept dimensions, in addition to the disciplinary core ideas.
Project Activities: The researchers will develop and validate multiple-choice and constructed-response assessment items organized into scenario-based tasks designed to measure students' three dimensional understanding of energy. Additional supporting materials for teachers, including the unpacking of the performance expectations, summaries of student misconceptions and difficulties, scoring rubrics, psychometric properties of the assessments, and guidelines for the use of the assessments and interpretation of results will also be developed.
Products: The products of this project include sets of NGSS aligned assessments for measuring students' three dimensional understanding of energy for use in elementary, middle and high school grade levels, along with supporting materials for teachers. All products will be made available on a website which will include an online testing utility. Researchers will also produce peer reviewed publications.
Setting: The setting for this study includes elementary, middle, and high schools from urban, rural, and suburban locations across the U.S.
Sample: The sample will include teachers and students in grades 4 through 12 across a range of ethnicities and socioeconomic status from across the U.S.
Intervention/Assessment: The researchers will develop and validate sets of three-dimensional assessment tasks aligned to elementary, middle, and high school Next Generation Science Standards (NGSS) performance expectations related to energy concepts. Specifically, the researchers will design tasks to assess student progress on: (1) transfer of energy by forces and conservation of energy, (2) thermal energy transfer and dissipation, and (3) energy and chemical reactions. The researchers will develop and investigate progressions along each dimension within these themes, including growth along the science and engineering practices dimension and the crosscutting concepts dimension. The researchers will develop 27 scenario-based assessment tasks that are each composed of 4–6 constructed-response and multiple-choice items. Each set or cluster of items will be built around an energy theme and real-world phenomenon that is appropriate for the students in the target grade band. The items will be designed not only to test for the correct scientific understanding but also to probe for common student misconceptions about energy and common difficulties students have with using science practices.
In addition, a professional development workshop will be developed to help support assessment developers and classroom teachers in creating their own three-dimensional assessments. To increase the practical utility of the assessment tasks, the researchers will produce supporting materials to guide the use of the tasks and the interpretation of the results.
Research Design and Methods: To develop the assessment, researchers will begin by selecting and clarifying sets of NGSS performance expectations (PEs) related to elementary, middle, and high school energy ideas. The PEs will define the specific practices, crosscutting concepts, and core ideas to be assessed. Then, three-dimensional assessment tasks will be developed using the PE clarification statements as item writing specifications. The tasks will be built around real-world scenarios and will be made up of a number of multiple-choice and constructed-response items. Some items will tap a single dimension, some will tap two, and some will tap all three, but the clustered sets of items will ensure that all three dimensions are assessed. During item development, 8–10 individual items will be written for each scenario. This will produce approximately 250 items distributed across the various themes and scenarios. From these, approximately 4–6 items will be selected to construct each of the final assessment tasks.
In Year 1, approximately 100 science teachers in grades 4 through 12 will be recruited to participate with their students in a pilot test of the tasks. It is anticipated that 100–200 students will respond to each assessment task during the pilot testing. Following pilot testing, the items, rubrics, and clarification statements will be formally analyzed by a panel of external expert reviewers. In Year 2, the assessment tasks will be field tested with a large national sample of students (1000–2000 students per item). Approximately 200 teachers and their students will be recruited to participate. In Year 3, the researchers will conduct a study of the assessments tasks to compare student responses using the multiple-choice items versus the constructed-response format. Approximately 1,000 students will be randomly assigned to each format. These students will also respond to a common set of assessments so that the properties of the assessment formats can be compared. Finally, in Year 4, a usability study will be conducted with 10 teachers and their students across grades 4–12 to determine the feasibility of using the tasks in a classroom setting and whether teachers use the information obtained from the tasks as intended (i.e., for improving instruction). The teachers will be asked to administer at least three out of the nine grade-level appropriate tasks (one from each theme) to their students and then use the scoring guides to analyze their results. Afterwards, teachers will complete an online survey about their experience using the tasks and support materials and will be interviewed by a member of the research team.
Control Condition: Not applicable for this study.
Key Measures: The key measures for this study include students' written responses to constructed response items, students' answer choice selections to multiple choice items, students' responses during interviews, reviewer feedback on draft items, and teachers' feedback on the supporting materials.
Data Analytic Strategy: The project will include qualitative analyses of students' written comments and interview responses and Rasch modeling of pilot and field test data. During field testing, researchers will use Wright maps to ensure that the assessment tasks are targeting the ability levels of the different grade bands. The research team will also use Differential item functioning to examine differences in the manner in which the assessment tasks defining the construct differs as a function of a subgroup (for instance, gender or whether English is the students' primary language).
Related IES Project: The Development and Validation of an Assessment Instrument to Study the Progression of Understanding of Ideas about Energy from Elementary School through High School (R305A120138)