Project Activities
The researchers will develop and validate three vertically aligned forms of an assessment instrument focusing on the topic of energy. In addition, the researchers will explore the progression of students' understanding of energy from late elementary through high school using the assessment. Supporting teacher materials will also be developed including descriptions of the targeted knowledge, summaries of student misconceptions about the targeted knowledge, psychometric properties of the assessment items, and guidelines using the instrument and interpretation of results.
Structured Abstract
Setting
The study will be conducted in elementary, middle, and high schools from urban, rural, and suburban areas across the United States.
Sample
The study will consist of a diverse, national sample of teachers and students from grades 4 through 12. The initial versions of the instrument will be pilot tested with approximately 3,000 students from120 classrooms. Once the finalized instrument is developed, it will then be administered to approximately 12,000 students from 480 classrooms.
Assessment
Each of the three forms of the assessment—one for late elementary grades (i.e., Grades 4 and 5), one for middle school, and one for high school—will include 35 multiple choice items and will be developed to provide a comprehensive measure of students' understanding of energy. The forms will be linked by overlapping items and by common conceptual themes that run through the three forms. The instruments will build on and extend the existing knowledge base in energy assessment by: (1) covering a more extensive and coherent set of important energy concepts than existing instruments; (2) testing for the correct scientific understanding and probing for common student misconceptions; (3) testing learning progressions through the energy topic; (4) making the instrument available in both paper-based and online formats; and (5) providing supporting teacher materials that will guide the interpretation of classroom- and student-level results.
Research design and methods
Four steps will be taken to develop and validate the instrument. First, the researchers will thoroughly describe the content domain that the instrument will assess and write clarification statements that unpack national content standards related to energy including those from the College Board, the National Research Council, and Achieve, Inc., to make explicit the boundaries around the knowledge being targeted. Second, an instrument item pool of 400 items will be developed. Once the item pool has been created, the third step is to pilot test the initial version of the instrument in 120 classrooms on a sample of approximately 1,000 students for each assessment form (3,000 students in all). The online version of the instrument will be administered to half of the students and the paper version will be administered to the other half. Once the instrument is revised and finalized, it will then be administered to a large national sample of late elementary, middle, and high school students in 480 classrooms with approximately 4,000 students for each assessment form (12,000 students in all). Teachers will be recruited through e-mail listservs to participate with their students. Teachers will be accepted into the study on a first-come, first-served basis, although the researchers will adjust the selection if necessary to achieve representation from urban, rural, and suburban schools from different parts of the country. Half the sample will use the online version, and the other half will use the paper version. These data will be used to examine the usability and validity of the online version of the assessment compared to paper-and-pencil versions, and to examine the progression of students' understanding of ideas about energy across grade levels.
Control condition
Due to the nature of this research design, there is no control condition.
Key measures
The key measures for the study include students' written responses to questions about pilot test items, students' responses during the Interviews about Instances and Events measure, students' answer choice selection during pilot and field tests of the assessment instrument, and teachers' feedback on the supporting materials.
Data analytic strategy
Qualitative analyses of students' written comments, interview responses, and Rasch modeling, will be used to construct and validate the three forms of the assessment and ensure the comparability of the online and paper-based versions of the assessment. Differential Item and Test Functioning Analysis will also be conducted to determine if any items or tests are sensitive to gender or English language learners.
People and institutions involved
IES program contact(s)
Products and publications
Products: Products include three fully developed and validated vertically aligned forms of the multiple choice assessment instrument and supporting materials for use in late elementary through high school. Peer-reviewed publications will also be produced.
Book chapter
Herrmann-Abell, C.F., and DeBoer, G.E. (2014). Developing and Using Distractor-Driven Multiple-Choice Assessments Aligned to Ideas About Energy Forms, Transformation, Transfer, and Conservation. In R.F. Chen, A. Eisenkraft, D. Fortus, J. Krajcik, K. Neumann, J.C. Nordine, and A. Scheff (Eds.), Teaching and Learning of Energy in K‐12 Education (pp. 103-133). New York: Springer.
Journal article, monograph, or newsletter
Herrmann-Abell, C.F., and DeBoer, G.E. (2017). Investigating a Learning Progression for Energy Ideas from Upper Elementary Through High School. Journal of Research in Science Teaching, 55 : 68-93. doi:10.1002/tea.21411
Supplemental information
Co-Principal Investigator: George DeBoer
Questions about this project?
To answer additional questions about this project or provide feedback, please contact the program officer.
