Grant Open

Improving Software and Methods for Estimating Diagnostic Classification Models and Evaluating Model Fit

NCER
Program: Statistical and Research Methodology in Education
Program topic(s): Early Career
Award amount: $224,996
Principal investigator: William Thompson
Awardee:
Virginia Commonwealth University
Year: 2021
Project type:
Methodological Innovation
Award number: R305D210045

Purpose

The goals of this project are to develop new software for the estimation and evaluation of the log-linear cognitive diagnosis model (the R package measr), and to use the software to conduct a simulation study evaluating the efficacy and efficiency of different model fit measures under a variety of conditions. Diagnostic classification models (DCMs) have shown the ability to provide more fine-grained and actionable scores to guide instruction and detect intervention effects. Despite these benefits, DCMs have yet to gain widespread use in applied research or operational settings. Most existing methods are either limited in their ability to fully assess absolute and relative model fit or are under-researched in the context of DCMs.

Project Activities

In the first stage of the project, the research team will develop measr. The software will estimate the linear cognitive diagnosis model by interfacing with the Stan programming language. This will allow for the implementation of a powerful estimator in a user-friendly interface that is free and available in an open-source environment. The software will also include functions for evaluating the absolute and relative fit of estimated models. Once developed, the research team will use the software in the second stage of the project during which they will conduct Monte Carlo simulations to determine the efficiency and efficacy of various measures of absolute and relative model fit that have been proposed for evaluating DCMs. They will refine the software per the simulation results and based on user-testing that will occur at multiple points during software development.

Products and publications

Products: The grant team will publish their findings and software developments in peer-reviewed journals and provide training workshops at a national conference and at Stats Camp to facilitate dissemination and use of measr.

Publications:

Thompson, W. J. (2023). measr: Bayesian psychometric measurement using Stan. Journal of Open Source Software, 8(91), 5742.

Questions about this project?

To answer additional questions about this project or provide feedback, please contact the program officer.

 

Questions about this project?

To answer additional questions about this project or provide feedback, please contact the program officer.