A decision-only judge matches GPT-6 on routine evals for 0.36% of the fee

For most LLM-as-a-judge work, you don’t need a frontier model writing out its reasoning. A Carnegie Mellon team compared TypeSafe JEV, a hosted judge that returns only a verdict plus label probabilities, against 16 generative and reward-model judges. On ordinary preference and evidence-grounded factuality it lands within 3 points of GPT-6 at 0.36% of the fee. On hard correctness checks it falls far behind. Its confidence score is reliable enough to tell you which case you’re in, so you can run it first and escalate only when it’s unsure.

• $0.044 per 1,000 judgments vs. $12.18 for GPT-6 (277x cheaper), and 0.152s median latency vs. 1.885s
• Preference (RewardBench): 92.2% vs. 93.5%. Factuality (HaluEval): 87.5% vs. 86.7%.
• Hard correctness (JudgeBench): 78.6% vs. 93.1%, a 14.6-point gap. Well-written wrong answers widen it to 19.8.
• Cascade: 99.6% of GPT-6’s accuracy at 47% of its fee, escalating 34% of items to GPT-6
• When JEV reports probability 1.0, it’s right 99.1% of the time (322 items)

Where the cheap judge holds up, and where it breaks

The study ran JEV 1.13 against 13 hosted judges (GPT-4.1 mini through GPT-6 Astra, Claude Sonnet 5, Gemini 3 Flash and 3.1 Pro, several Qwen models) and 4 local ones (including PairRM and Skywork-Reward-V2). Disagreements between judges went to blinded human adjudication. The price gap is the headline:

Accuracy depends on the workload. The paper sorts each one into “use JEV” or “escalate” based on the gap to GPT-6:

The three blue gaps all have 95% confidence intervals that cross zero. The red ones don’t come close (JudgeBench: −18.9 to −10.3). The pattern makes sense for a judge that doesn’t reason out loud. It can tell which answer is better supported or more helpful, but it can’t re-derive a proof or a calculation to catch a subtle error. It is also easily fooled by a wrong answer that is elaborately written. GPT-6 fixes 60 of JEV’s 75 JudgeBench errors; JEV fixes only 9 of GPT-6’s 24.

The confidence score is the useful part

A cheap judge is only safe if you know when to distrust it. JEV returns a probability for each label, and higher confidence reliably means higher accuracy across the pooled tasks:

That makes a simple cascade work: accept JEV’s verdict when q ≥ τ, otherwise send the item to GPT-6. At τ = 0.9, the cascade escalates 34% of items and scores 91.3% vs. GPT-6’s 91.7% at 47% of GPT-6’s fee (about $6.30 per 1,000 judgments). How much it saves depends on the workload. On RewardBench it actually beats GPT-6 (94.0% vs. 93.5%) at 22% of the fee. On JudgeBench it escalates 61% of items and still costs 62% of the fee.

For preference pairs, the authors also judge both orderings (A,B) and (B,A) and average the probabilities, since JEV flips its answer on 3.25% of RewardBench pairs and 11.14% of JudgeBench pairs when the order is swapped. They froze these two-order policies and tested them on 510 held-out pairs. With GPT-6 as fallback, τ = 0.9 kept JEV’s verdict on 53.7% of items and scored 92.5% vs. 93.1% at 57% of the fee. With GPT-5.6 Sol as fallback, though, the threshold picked earlier didn’t transfer: the cascade lost 2.4 points, and that gap is significant. The takeaway: pick τ per fallback on your own data, then re-check it.

Caveats

• Confidence isn’t a certificate. On style-adversarial pairs, the confidence signal degrades (AUROC 0.770). Worse, when HaluEval is judged without the source document, every judge is near chance (JEV 52.5%, GPT-5.4 55.0%), yet mean confidence stays at 0.90 to 0.96. A cascade can’t catch errors the judge doesn’t know it’s making.
• Calibration doesn’t transfer. No single temperature fits every workload (fitted values ranged from 0.65 to 4.45), and applying a pilot-fitted one to held-out data made RewardBench and JudgeBench calibration worse.
• Labels are noisy. 24 of the 26 HaluEval items that both judges missed turned out to have unsupported gold labels; corrected scores are 95.8% (JEV) and 98.3% (GPT-6). Human review also widened GPT-6’s RewardBench lead to 3.0 points. But all 183 adjudicated items were labeled by a single annotator, and RewardBench agreement was low (κ = 0.29).
• It’s one proprietary product. JEV is a single version of a commercial service, with unknown training data and benchmark contamination. The comparison ignores architecture, model size, and reasoning effort, and the cascade fees are simulated from API prices, not invoices. Specialized professional domains weren’t tested.

The practical rule holds regardless of vendor: if your evals are mostly “which of these is better” or “is this supported by the document,” a cheap decision-only judge with a confidence gate gets you most of a frontier judge’s accuracy for a fraction of the cost. If they involve checking math, code, or derivations, pay for the stronger judge.