|Title:||Establishing the Validity and Diagnostic Capacity of Facet-Based Science Assessments|
|Principal Investigator:||DeBarger, Angela||Awardee:||SRI International|
|Program:||Science, Technology, Engineering, and Mathematics (STEM) Education [Program Details]|
|Award Period:||4 years||Award Amount:||$1,819,505|
Co-Principal Investigators: Louis DiBello, James Minstrell
Purpose: Formative use of diagnostic classroom assessments can be one of the most powerful ways to improve student achievement. These assessments have the potential to provide critical information to students and teachers about whether students understand the targeted concepts, and if not, what problematic or partial understandings are present instead. Facet-based assessments are one innovative approach to help teachers diagnose students' science understanding. A facet is the use of one or more pieces of knowledge and reasoning by the learner in order to solve a problem or explain an event. This study will focus on the development and validation of facet-based assessments designed to improve science learning and instruction in Force and Motion units.
Project: The researchers will design, validate, and revise assessments focusing on 17 facet clusters related to three key concept strands in Force and Motion at the middle and high school level—Description of Motion; Nature of Forces; and Forces to Explain Motion.
Products: The outcomes of the project include facet-based diagnostic assessments that teachers can use to improve science learning and instruction, along with published reports.
Setting: The study will be conducted in middle and high schools in Washington State.
Population: The study sample will consist of 40 middle and high school physical science and physics classrooms. The schools include diverse student populations, with up to 50 percent of students qualifying for free- or reduced-price lunch.
Intervention: Students' thinking can be organized into hypothetical facet clusters around a key idea or event. The facet clusters serve as the interpretive framework for analysis of student responses to assessment items and for adapting curricular activities and instruction to promote learning. This project will focus on assessments associated with 17 facet clusters related to three key concept strands in Force and Motion at the middle and high school level—Description of Motion; Nature of Forces; and Forces to Explain Motion. Sets of diagnostic questions aligned with each facet cluster are presented to students and teachers online via Diagnoser.com. The project focuses on student responses to more than 300 questions in a total of 31 sets of questions associated with the 17 facet clusters.
Research Design and Methods: The study includes three phases: (1) Analyses of existing sets of assessment items and associated data; (2) Assessment refinement and new assessment development; and (3) Collection of new data, and evaluation of refined and new assessment items and sets of items. In Phase I, the researchers will conduct advanced psychometric analyses to examine the cognitive, instructional, and empirical validity of existing facet-based assessments. In Phase II, the evidence-centered design (ECD) framework will be used to provide a systematic procedure for integrating existing facet-based design, findings on cognitive and instructional validity, and advanced psychometric and statistical data modeling. ECD will also serve to promote the redesign of existing items, or the development of new items that will be incorporated into the question sets. Up to 17 task templates will be created. In Phase III, the researchers will focus on conducting alignment studies and construct validity studies on the new and revised items and sets. Items will be administered to students via the online Diagnoser.com system. The researchers expect to be able to gather data from at least 20 classrooms per year, over the last 2 years of the project, totaling 40 classrooms with 600 students.
Control Condition: There is no control condition.
Key Measures: The key measures for the study include students' responses on the diagnostic assessments, classroom observations, logs, and student and teacher interviews.
Data Analytic Strategy: Multivariate psychometric models will be used to analyze the diagnostic assessments. Analyses include a type of constrained latent class model called the Fusion Model, model calibration using Markov Chain Monte Carlo methods, and two-level hierarchical linear modeling to account for the clustering of students within classrooms.
Project Website: http://ctl.sri.com/projects/displayProject.jsp?Nick=facet
Haertel, G.D., Vendlinski, T.P., Rutstein, D., DeBarger, A., Cheng, B.H., Snow, E.B., D'Angelo, C., Harris, C., Yarnall, L., and Ructtinger, L. (2016). General Introduction to Evidence-Centered Design. In H.I. Braun (Ed.), Meeting the Challenges to Measurement in an era of Accountability (pp. 107–148). New York: Routledge.
Journal article, monograph, or newsletter
DiBello, L.V., Henson, R.A., and Stout, W.F. (2015). A Family of Generalized Diagnostic Classification Models for Multiple Choice Option-Based Scoring. Applied Psychological Measurement, 39 (1): 62–79.
Pellegrino, J.W., DiBello, L.V., and Goldman, S.R. (2016). A Framework for Conceptualizing and Evaluating the Validity of Instructionally Relevant Assessments. Educational Psychologist, 51 (1): 59–81.
Nongovernment report, issue brief, or practice guide
Fujii, R., Haertel, G., McElhaney, K., D'Angelo, C., Werner, A., Ructtinger, L., Feng, M., Gong, B., and DeBarger, A. (2015). The Performance of Facet-Based Items: A Cognitive Analysis Study, Technical Report II. Menlo Park, CA: SRI International.
DiBello, L.V., Henson, R.A., and Stout, W.F. (in press). Enhanced Reparameterized Unified Model: A Diagnostic Classification Model for Multiple Choice Option-Based Scoring. In Proceedings of the International on Achievement Assessment and Evaluation . Shanghai, China.