Grant Closed

Increased Accuracy in the Detection of Differential Item Functioning through Multilevel Analysis

NCER

Program: Statistical and Research Methodology in Education

Program topic(s): Core

Award amount: $453,933

Principal investigator: Brian French

Awardee:

Washington State University

Year: 2011

Award period: 1 year 7 months (06/01/2013 - 12/31/2014)

Project type:

Methodological Innovation

Award number: R305D110014

Purpose

The purpose of the research project was to create multilevel differential item functioning (DIF) methods and software to increase the accuracy of the detection of DIF. Detection of DIF is one step in the process of gathering score validity evidence. DIF occurs when examinees from different subgroups who have equal ability on the measured construct have different probabilities of responding correctly to an item. DIF detection is a common component of instrument development in testing programs with high standards for the psychometric properties of tests.

Two popular methods for DIF detection are SIBTEST and the Mantel-Haenszel (MH) statistic. These methods have proven to be effective at detecting DIF when present, while maintaining control of the Type I error rate. MH and SIBTEST have both been shown to be especially effective with small samples, particularly when compared with more complex, model-based approaches.

Neither method has been adapted for use with multilevel data and thus may exhibit poor performance when such a data structure is present. Multilevel data is prevalent in DIF studies (e.g., examinees sampled within classrooms which are sampled from within schools). If standard analyses at the individual level are used ignoring the multilevel structure, variation due to higher levels of the data structure (e.g., classrooms) could be conflated with that due to student level variations. This can lead to incorrect parameter estimates and standard errors thereby degrading DIF detection.

Project Activities

This project created the appropriate multilevel versions of SIBTEST and the MH statistic for both dichotomous and polytomous items. In addition, an extension of SIBTEST, known as crossing SIBTEST, which is used to detect nonuniform DIF (the case where an item discriminates across the levels of ability differently for different groups), was developed for use with multilevel data. In addition to developing the statistics to accommodate multilevel data, the project also developed code in R and SAS that can be used to conduct the DIF analyses. The performance of the techniques developed will be evaluated in terms of power and Type I errors for DIF detection through Monte Carlo simulation studies and the analysis of existing data sets.

People and institutions involved

IES program contact(s)

Allen Ruby

Project contributors

W. Holms Finch

Co-principal investigator

Products and publications

Book chapter

French, B. F., and Finch, W. H (2016). Detecting Differential Item Functioning. In K. Schweizer and C. DiStefano (Eds.), Principles and Methods of Test Construction (pp. 197-217). Hogrefe Publishing.

French, B. F., Finch, W. H., & Immekus, J. C. (2019, June). Multilevel generalized Mantel-Haenszel for differential item functioning detection. In Frontiers in Education (Vol. 4, p. 47). Frontiers Media SA.

Journal article, monograph, or newsletter

Beaver, J., French, B. F., Finch, W. H., and Ullrich-French, S. C. (2014). Sex Differential Item Functioning in the Inventory of Early Development III Social-Emotional Skills. Journal of Psychoeducational Assessment, 32(8): 775-780.

French, B. F., and Finch, W. H. (2015). Transforming SIBTEST to Account for Multilevel Data Structures. Journal of Educational Measurement, 52(2): 159-180.

French, B.F., and Finch, W.H. (2013). Extensions of Mantel-Haenszel for Multilevel DIF Detection. Educational and Psychological Measurement, 73(4): 648-671.

French, B.F., Finch, W.H., Randel, B., Hand, B., and Gotch, C.M. (2016). Measurement Invariance Techniques to Enhance Measurement Sensitivity. International Journal of Quantitative Research in Education, 3(1-2), 79-93.

French, B.F., Finch, W.H., and Vazquez, J.A.V. (2016). Differential Item Functioning on Mathematics Items Using Multilevel SIBTEST. Psychological Test and Assessment Modeling, 58(3), 471-483.