Development of Accessible IRT-Based Models and Methodologies for Improving the Breadth and Accuracy of Item Option-Scored Diagnostic Assessments

The primary purpose of this research project is to fully develop and evaluate models and methodology for diagnostic assessment that can be applied to well-designed tests that incorporate multiple choice and/or short answer questions. Researchers anticipate continued extensive use of tests composed of multiple choice and/or short answer questions in next generation assessments. The use of these assessment types is based on the need to keep testing time down to reasonable levels, the relative efficiency of scoring and reporting assessment performance on multiple choice and short answer questions, and the measurement advantages of gathering multiple locally independent measurements per unit of testing time. Because of the pressures for continued use of multiple choice and short answer questions, the diagnostic power and efficiency of such assessments is crucial.

The research in this project is organized around a general psychometric family of diagnostic models called the Generalized Diagnostic Classification Model for Option-Based Multiple Choice Scoring (GDCM-MC). Researchers on this project will investigate the performance of this family of models on data from five tests that were developed for the purpose of diagnosis and on simulated datasets. Products of this research project will include user-friendly open source software and an accompanying manual for conducting analyses with the developed models, model comparison, and model selection guidelines for researchers and practitioners. The team will also prepare guidelines for planning and implementing diagnostic tests for classroom and large scale use.