An LLM judge gave the same verdict 100% of the time and was right 51% of the time

Determinism is not reliability. In a new audit of six LLM judges, Gemini 2.0 Flash returned the identical verdict on every one of 10 repeated calls, yet agreed with ground truth on only 51% of JudgeBench pairs. The reason: swapping which answer was shown first flipped 98% of its verdicts. It was consistently picking a slot, not an answer.

• 98% of verdicts flipped under an order swap in the worst cell (Gemini 2.0 Flash, verdict-first XML, JudgeBench); its trustworthy verdict rate was 2%
• 7 to 17% of verdicts changed across identical reruns at temperature zero
• Output format alone moved Gemini 2.5 Pro from 28.9% to 77.5% trustworthy verdicts on the same benchmark
• No universal best judge: the judge × dataset interaction explained more variance (28.1%) than the judge (27.0%) or the dataset (26.2%)
• Single-prompt rubric scoring (G-Eval) beat four-call rubric voting (FLASK), which lowered trustworthiness for all six judges on LLMBar

Accuracy hides two failure modes

The researchers ran 400 preference pairs (100 each from MT-Bench, AlpacaFarm, JudgeBench, and LLMBar) through GPT-5.2, GPT-4o, Gemini 2.5 Pro, Gemini 2.0 Flash, Claude Sonnet 4.6, and Llama 3.3 70B. Every pair went through five output formats, both presentation orders, and 10 repetitions, at up to three temperatures. Total API bill: about $2,100.

For each cell they measured three things:

• Consistency (C): share of pairs where all 10 reruns agree
• Flip rate (φ): share of pairs where the verdict changes when the two answers swap positions
• Accuracy (A): agreement with the ground-truth label

The flip rate puts a hard ceiling on accuracy. With a forced binary choice, a pair that flips gets one right answer and one wrong answer across the two orders, so it scores exactly 50%. That means accuracy can never exceed 1 − φ/2, no matter how capable the model is. The 51% judge sits exactly on that line: 98% of pairs at 50%, plus 2% of pairs it got right in both orders.

The paper combines the three into a trustworthy verdict rate, T = C × (A − φ/2): the share of verdicts that are reproducible, order-invariant, and correct. Subtracting φ/2 strips out the accuracy a judge gets for free on pairs where it is effectively flipping a coin.

The metric also separates judges that accuracy lumps together. On AlpacaFarm, GPT-5.2 with CoT JSON scored 57.7% accuracy and Claude Sonnet 4.6 with direct prompting scored 54.7%. But GPT-5.2 flipped twice as often (15.5% vs 7.4%), so Claude had the higher trustworthy rate: 49.9% vs 40.2%.

Format matters as much as model choice

Changing only the output instruction (single-token verdict, verdict-then-rationale, or rationale-then-verdict, in JSON or XML) produced some of the largest swings in the study. Gemini 2.5 Pro on JudgeBench is the extreme case: its flip rate dropped from 56.0% under direct prompting to 6.9% under verdict-first JSON.

There is no format that wins everywhere. Verdict-first JSON was Gemini 2.5 Pro’s best format on JudgeBench and Gemini 2.0 Flash’s second worst (10.8%), just above verdict-first XML. Chain of thought hurt GPT-4o on every dataset, raising its JudgeBench flip rate from 24% to 47%. GPT-5.2 and Llama 3.3 barely moved with format; the two Gemini models swung by about 20 to 25 points on average. Claude Sonnet 4.6 had the highest mean trustworthy rate (62.8% across 20 cells), but its own range ran from 48.9% to 81.0%.

Temperature behaved the way the formula predicts. Raising it from 0 to 1.0 cut mean consistency by 12.1 points while leaving bias-adjusted accuracy almost unchanged (+0.3 points). Noise and position bias are separate problems with separate fixes.

What to do with your own judge

The paper sorts judges into four regimes by consistency and flip rate, and each needs a different fix:

• Noisy but fair (low C, low φ): sample several times and take a majority vote
• Confidently biased (high C, high φ): the dangerous one, since reruns look stable. Run both orders and only count verdicts that agree
• Chaos (low C, high φ): do both, average across orders and across reruns
• Trustworthy (high C, low φ): the remaining gap is capability; you need a better judge

On the two objective benchmarks, the authors also tested rubric scoring. FLASK, which splits judgment into four separate calls and votes, cut GPT-5.2’s consistency from 86.0% to 34.8% on JudgeBench and lowered trustworthiness for every judge on LLMBar and half of them on JudgeBench. G-Eval, which scores four criteria in a single prompt, mostly preserved consistency, and for the stronger judges it cut flips sharply (it made GPT-4o and Llama 3.3 flip more). For Gemini 2.5 Pro on JudgeBench it raised T from 28.9% to 77.9%, more than any format change.

The cheapest lesson needs no new metric: if your eval only runs each pair in one order, you cannot see φ at all, and you have no idea whether your judge is reading the answers or the slots. Running both orders doubles the cost and immediately tells you your accuracy ceiling. (For the opposite end of the cost curve, see our writeup on a decision-only judge that matched GPT-6 for 0.36% of the fee.)

Caveats

• Small samples. Each benchmark contributed 100 pairs, so a single cell’s numbers carry real sampling noise. The large effects (98% flips, 48.6-point format swings) are hard to explain away; differences of a few points are not.
• The judge lineup is uneven. GPT-4o, Gemini 2.0 Flash, and Llama 3.3 sit alongside current models. The worst cells come from older models, and results may not carry over to newer judges.
• The ceiling is arithmetic, not a discovery. A ≤ 1 − φ/2 follows directly from forced binary choice. Its value is that it makes position bias show up as lost accuracy, and it changes once ties or “neither” verdicts are allowed.
• The closed form assumes independence between reproducibility, order invariance, and correctness. The authors check it against the directly measured joint rate and find the gap is usually under 5 points, with the formula slightly underestimating.
• Consistency is a strict bar. C counts a pair as consistent only if all 10 reruns agree, so it penalizes judges that are right 9 times in 10.
• Rubric results cover only JudgeBench and LLMBar, and only pairwise comparisons; listwise ranking and absolute grading were not tested.
• Version 1, released October 8. No independent replication yet.

The recommendation is easy to adopt: report consistency, flip rate, accuracy, and T per judge, format, and dataset, and stop treating single-order evaluation as a complete protocol. If a judge’s output feeds a reward model or a leaderboard, a 2% trustworthy rate behind a 100% consistency score is exactly the kind of failure nobody notices until it has shaped the results.