The researchers will design, validate, and revise assessments focusing on 17 facet clusters related to three key concept strands in Force and Motion at the middle and high school level—Description of Motion; Nature of Forces; and Forces to Explain Motion.
Setting
The study will be conducted in middle and high schools in Washington State.
Sample
The study sample will consist of 40 middle and high school physical science and physics classrooms. The schools include diverse student populations, with up to 50 percent of students qualifying for free- or reduced-price lunch.
The study includes three phases: (1) Analyses of existing sets of assessment items and associated data; (2) Assessment refinement and new assessment development; and (3) Collection of new data, and evaluation of refined and new assessment items and sets of items. In Phase I, the researchers will conduct advanced psychometric analyses to examine the cognitive, instructional, and empirical validity of existing facet-based assessments. In Phase II, the evidence-centered design (ECD) framework will be used to provide a systematic procedure for integrating existing facet-based design, findings on cognitive and instructional validity, and advanced psychometric and statistical data modeling. ECD will also serve to promote the redesign of existing items, or the development of new items that will be incorporated into the question sets. Up to 17 task templates will be created. In Phase III, the researchers will focus on conducting alignment studies and construct validity studies on the new and revised items and sets. Items will be administered to students via the online Diagnoser.com system. The researchers expect to be able to gather data from at least 20 classrooms per year, over the last 2 years of the project, totaling 40 classrooms with 600 students.
Control condition
There is no control condition.
Key measures
The key measures for the study include students' responses on the diagnostic assessments, classroom observations, logs, and student and teacher interviews.
Data analytic strategy
Multivariate psychometric models will be used to analyze the diagnostic assessments. Analyses include a type of constrained latent class model called the Fusion Model, model calibration using Markov Chain Monte Carlo methods, and two-level hierarchical linear modeling to account for the clustering of students within classrooms.
Products: The outcomes of the project include facet-based diagnostic assessments that teachers can use to improve science learning and instruction, along with published reports.
Book chapter
Haertel, G.D., Vendlinski, T.P., Rutstein, D., DeBarger, A., Cheng, B.H., Snow, E.B., D'Angelo, C., Harris, C., Yarnall, L., and Ructtinger, L. (2016). General Introduction to Evidence-Centered Design. In H.I. Braun (Ed.), Meeting the Challenges to Measurement in an era of Accountability (pp. 107-148). New York: Routledge.
Journal article, monograph, or newsletter
DiBello, L.V., Henson, R.A., and Stout, W.F. (2015). A Family of Generalized Diagnostic Classification Models for Multiple Choice Option-Based Scoring. Applied Psychological Measurement, 39 (1): 62-79.
Pellegrino, J.W., DiBello, L.V., and Goldman, S.R. (2016). A Framework for Conceptualizing and Evaluating the Validity of Instructionally Relevant Assessments. Educational Psychologist, 51 (1): 59-81.
Nongovernment report, issue brief, or practice guide
Fujii, R., Haertel, G., McElhaney, K., D'Angelo, C., Werner, A., Ructtinger, L., Feng, M., Gong, B., and DeBarger, A. (2015). The Performance of Facet-Based Items: A Cognitive Analysis Study, Technical Report II. Menlo Park, CA: SRI International.
Proceeding
DiBello, L.V., Henson, R.A., and Stout, W.F. (in press). Enhanced Reparameterized Unified Model: A Diagnostic Classification Model for Multiple Choice Option-Based Scoring. In Proceedings of the International on Achievement Assessment and Evaluation . Shanghai, China.
Co-Principal Investigators: Louis DiBello, James Minstrell
