Differential Item Functioning: Detecting Item Bias

Dec 12, 2025 min read
Differential Item Functioning: Detecting Item Bias

Every test makes a quiet promise: two people with the same ability should have the same chance of getting an item right, no matter which group they belong to. Differential item functioning (DIF) is what we call it when an item breaks that promise.

This post is the version of DIF I wish I had when I started: what it is, what it is not, the methods people actually use, and what to put in the paper so the analysis is defensible.

What DIF is (and is not)

An item shows DIF when examinees from different groups, matched on the trait being measured, have different probabilities of a given response. The matching is the whole point.

  • DIF is not a group difference. If one group scores lower on an item because it scores lower on the trait, that is impact, not DIF. Impact is about the people; DIF is about the item.
  • DIF is not automatically bias. DIF is a statistical flag. Bias is a judgment that the difference comes from something irrelevant to the construct, like unfamiliar vocabulary, cultural context, or a translation that changed the meaning. You need content review to get from one to the other.
  • DIF is item-level. Measurement invariance asks the same question at the level of the whole measurement model. The two ideas are close cousins, and I wrote a separate invariance checklist for that side.

Uniform vs non-uniform DIF

The picture at the top of this post shows the two flavors.

  • Uniform DIF: one group has an advantage at every ability level. In IRT terms, the difficulty (location) parameter differs, and the item characteristic curves are shifted but parallel.
  • Non-uniform DIF: the advantage depends on ability, so the curves cross. In IRT terms, the discrimination (slope) parameter differs. An item might favor the focal group at low ability and the reference group at high ability.

This matters because some methods only look for uniform DIF. If you never test for non-uniform DIF, you cannot claim you did not find it.

The main methods

Mantel-Haenszel

The classic. Examinees are stratified by total score, a 2 × 2 table (group × correct) is built within each stratum, and a common odds ratio is estimated across strata. It is fast, works with modest samples, and has a well-known effect size: the ETS delta scale, ΔMH = -2.35 × ln(αMH).

ETS classification is the usual reporting shorthand:

  • A (negligible): |ΔMH| < 1, or not significantly different from 0.
  • C (large): |ΔMH| ≥ 1.5 and significantly greater than 1.
  • B (moderate): everything in between.

The catch: Mantel-Haenszel is built for uniform DIF. It can miss crossing curves almost completely.

Logistic regression

Logistic regression (Swaminathan & Rogers, 1990) fits three nested models for each item:

Model 1:  logit(P) = b0 + b1*score
Model 2:  logit(P) = b0 + b1*score + b2*group                  # uniform DIF
Model 3:  logit(P) = b0 + b1*score + b2*group + b3*score*group # non-uniform DIF

Comparing Model 3 to Model 1 tests for any DIF; Model 2 vs 1 isolates uniform DIF, and Model 3 vs 2 isolates non-uniform DIF. The ordinal version handles Likert items. For effect size, use the change in pseudo R² between models. A common rule (Jodoin & Gierl, 2001) treats ΔR² below .035 as negligible, .035 to .070 as moderate, and above .070 as large.

IRT-based approaches

If you already fit an IRT model, the likelihood ratio approach is the natural one: fit a multiple-group model with the studied item's parameters constrained equal across groups, free them, and compare. Because you compare parameters directly, you get uniform and non-uniform DIF in the same framework, and you can report the size of the difference on the item's own scale. The cost is that you need larger samples and a model that actually fits.

MIMIC models

In SEM, a MIMIC model regresses the latent factor on group and then adds a direct path from group to an item. A significant direct effect means the item responds to group membership beyond what the factor explains, which is uniform DIF. It is convenient when you want covariates or several grouping variables at once, but the standard version does not test non-uniform DIF.

The anchor problem (the part people skip)

Every DIF method compares groups against something assumed to be DIF-free: the total score, or a set of anchor items. If the matching criterion itself contains DIF items, the matching is contaminated and you get false positives in one direction and missed DIF in the other.

Two habits help a lot:

  • Purification: run the analysis, drop flagged items from the matching score, and rerun until the flagged set stops changing.
  • Deliberate anchors: in IRT-based DIF, pick anchor items that content experts and earlier analyses agree are clean, and say which ones you used.

Multiple testing

Test 40 items at α = .05 and you should expect about two flags by chance alone. With large samples, almost everything becomes significant too. Both problems have the same fix: control the error rate (Benjamini-Hochberg is a reasonable default) and require an effect size threshold before calling an item a DIF item. Significance says "detectable", effect size says "worth worrying about".

Quick R sketch

library(difR)
library(lordif)

# Mantel-Haenszel with purification (binary items)
mh <- difMH(Data = items, group = sex, focal.name = "F", purify = TRUE)

# Logistic regression, uniform + non-uniform, with purification
lr <- difLogistic(Data = items, group = sex, focal.name = "F",
                  type = "both", purify = TRUE)

# Ordinal logistic regression with IRT-based matching and an R^2 flag
ol <- lordif(resp.data = likert_items, group = sex,
             criterion = "R2", pseudo.R2 = "Nagelkerke", R2.change = 0.035)
summary(ol)

What to report

  • The grouping variable(s), sample size per group, and why these groups matter for the test's use.
  • The method, the matching criterion, and whether (and how) it was purified or anchored.
  • Both significance tests and effect sizes, with the classification rule you used.
  • How you handled multiple testing.
  • For flagged items: direction, uniform vs non-uniform, and the content review verdict.
  • What you did about it (kept, revised, removed, modeled) and whether conclusions changed.

Closing

DIF analysis is not a box to tick at the end of validation. It is one of the most direct ways to check the promise a test makes to the people taking it. Flag carefully, size the effects, and then do the part the software cannot: look at the item and ask why.


- Fatih