Inter-Rater Reliability

Inter-rater reliability measures how much agreement exists between independent raters (human or automated) beyond what random chance would produce. It answers: “if two people rate the same thing, do they agree — and is that agreement meaningful or just luck?”

Prerequisites

• Basic probability (what “expected by chance” means)

Why it matters

When multiple judges rate the same output (e.g., two humans scoring LLM responses, or a human and an LLM judge), raw percent agreement is misleading. If there are only two categories and both raters default to “good,” they’ll agree 50%+ of the time by accident. Inter-rater reliability corrects for this baseline.

Cohen’s Kappa ($\kappa$)

Designed for two raters on categorical labels (e.g., pass/fail, positive/negative/neutral).

$\kappa =\frac{p_o-p_e}{1-p_e}$

Notation

• $p_o$ — observed agreement (fraction of cases where both raters gave the same label)
• $p_e$ — expected agreement by chance (if each rater assigned labels randomly at their observed rates)

Interpretation:

$\kappa$	Meaning
< 0	Worse than chance (systematic disagreement)
0.0	Agreement equals chance
0.01 - 0.20	Slight agreement
0.21 - 0.40	Fair agreement
0.41 - 0.60	Moderate agreement
0.61 - 0.80	Substantial agreement
0.81 - 1.00	Almost perfect agreement

How to compute $p_e$ (expected agreement by chance)

$p_e$ asks: “if both raters were flipping weighted coins (using their individual label frequencies), how often would they land on the same label by accident?”

For each label $k$, compute the probability that both raters independently pick that label:

$p_e={\sum }_{k}P(\text{Rater A picks }k)\times P(\text{Rater B picks }k)$

Where:

• $P(\text{Rater A picks }k)=\frac{\text{number of times A used label }k}{\text{total items}}$
• $P(\text{Rater B picks }k)=\frac{\text{number of times B used label }k}{\text{total items}}$

In other words: look at how often each rater uses each label overall, then compute the probability they’d both land on the same label if they were rating independently (without even looking at the item).

Worked example

Two raters classify 100 LLM outputs as “good” or “bad”:

	Rater B: Good	Rater B: Bad	Total
Rater A: Good	40	10	50
Rater A: Bad	5	45	50
Total	45	55	100

Step 1 — Observed agreement ($p_o$):

Count the cases where both raters gave the same label (the diagonal cells): $40+45=85$ agreements out of 100.

$p_o=\frac{85}{100}=0.85$

Step 2 — Expected agreement by chance ($p_e$):

Look at each rater’s marginal frequencies (the row/column totals):

• Rater A said “Good” 50 times out of 100 → $P(A=\text{Good})=0.50$
• Rater A said “Bad” 50 times out of 100 → $P(A=\text{Bad})=0.50$
• Rater B said “Good” 45 times out of 100 → $P(B=\text{Good})=0.45$
• Rater B said “Bad” 55 times out of 100 → $P(B=\text{Bad})=0.55$

If they were rating independently (random coins with these weights):

$p_e=P(A=\text{Good})\times P(B=\text{Good})+P(A=\text{Bad})\times P(B=\text{Bad})$ $p_e=(0.50\times 0.45)+(0.50\times 0.55)=0.225+0.275=0.50$

So by pure chance, we’d expect 50% agreement.

Step 3 — Kappa:

$\kappa =\frac{p_o-p_e}{1-p_e}=\frac{0.85-0.50}{1-0.50}=\frac{0.35}{0.50}=0.70$

Interpretation: They agree 85% of the time, but chance alone would give 50%. Kappa = 0.70 means they captured 70% of the agreement that was available beyond chance — substantial agreement.

Limitations

• Only works for exactly two raters
• Assumes categorical (nominal) labels — doesn’t account for “how wrong” a disagreement is (rating 1 vs 5 counts the same as 4 vs 5)
• Can be paradoxically low when prevalence is very skewed (if 95% of cases are “good,” even high raw agreement produces low $\kappa$)

Krippendorff’s Alpha ($\alpha$)

A more general metric that handles:

• Any number of raters (not just two)
• Ordinal and continuous scales (not just categories) — a disagreement of 1 vs 5 counts more than 4 vs 5
• Missing data (raters don’t need to rate every item)

$\alpha =1-\frac{D_o}{D_e}$

Notation

• $D_o$ — observed disagreement (average distance between ratings on the same item)
• $D_e$ — expected disagreement by chance (average distance if ratings were randomly assigned)

Interpretation: Same scale as $\kappa$ — 0 is chance, 1 is perfect. Krippendorff himself recommends $\alpha \ge 0.667$ as the minimum for tentative conclusions and $\alpha \ge 0.8$ for reliable data.

When to use which

Situation	Use
Two raters, pass/fail labels	Cohen’s $\kappa$
Two raters, 1-5 scale	Weighted $\kappa$ (accounts for distance between ratings)
3+ raters, any scale	Krippendorff’s $\alpha$
Missing data (not all raters scored all items)	Krippendorff’s $\alpha$

Application to LLM evaluation

In Prompt Evaluation, inter-rater reliability shows up in two ways:

1. Human eval quality: If your human raters disagree ($\kappa <0.4$), the rubric is underspecified. Fix the rubric before collecting more labels.
2. Judge validation: Compare LLM judge scores against human scores. If $\kappa <0.6$ between the judge and humans, the judge isn’t reliable enough for that rubric — simplify the rubric or use a stronger judge model.

See also

• Prompt Evaluation — where inter-rater reliability is used in practice
• LLM as Judge — validating automated judges against human agreement