GPT-4 scored 86.4% on MMLU when it launched. Eighteen months later, every frontier model scores above 88%. And yet, the model I deployed last month still hallucinated a customer's billing address into a support ticket. The benchmark said it was smart. Production said otherwise.
I used to check leaderboards before picking a model. MMLU scores, HumanEval pass rates, GSM8K accuracy — the whole routine. I'd compare decimal points like they meant something. Then I shipped three LLM-powered products and learned that benchmark performance predicts production performance about as well as a resume predicts job performance. Loosely. Sometimes. If you squint.
Here's what changed my mind, and what I measure now instead.
Let me start with the data, because the data is damning.
Microsoft researchers built MMLU-CF, a contamination-free version of the MMLU benchmark. Same difficulty, same format — just questions the models hadn't seen during training. The result: top models' accuracy dropped by 14-16 points compared to the original MMLU. That's not a minor variance. That's the difference between "impressive" and "mediocre."
It gets worse. When researchers removed contaminated examples from GSM8K — the popular math benchmark — accuracy dropped by 13%. These models weren't reasoning through math problems. They were recalling answers from training data.
The Oxford Internet Institute analyzed 445 LLM benchmarks and found that only 16% used rigorous scientific methodology. About half claimed to measure abstract concepts like "reasoning" or "harmlessness" without ever defining what those words mean.
And then there's the cheating. When researchers analyzed 2.8 million comparison records from LMArena (formerly Chatbot Arena), they found that selective model submissions inflated scores by up to 100 Elo points. Meta, OpenAI, Google, and Amazon were privately testing multiple model variants and only publishing the best results.
This isn't evaluation. It's marketing.
Here's a question most benchmark articles won't ask: what happens when every model scores 90%+?
We're already there. Frontier models have saturated MMLU above 88%. GSM8K? The top models hit 99%. When GPT-5.3, Claude Opus 4.6, and Gemini 3.1 all score within a few points of each other, the benchmark tells you nothing about which one will work better for your specific use case.
The industry's response has been to build harder benchmarks. MMLU-Pro expanded to 12,000 graduate-level questions with ten answer choices instead of four. It caused accuracy drops of 16% to 33% compared to original MMLU. Problem solved, right?
MMLU-Pro was saturated by November 2025. Google's Gemini 3 Pro hit 90.1%. Eighteen months from "this will differentiate models" to "everyone scores the same." The treadmill keeps spinning.
GPQA Diamond was supposed to be the hard one — 198 questions where PhD experts only achieve 65% accuracy. ARC-AGI tests genuine abstraction that resists memorization. These are better benchmarks, genuinely. But the pattern is clear: every new benchmark has a shelf life. Create it, watch it get gamed, replace it. Repeat.
"When a measure becomes a target, it ceases to be a good measure."
Charles Goodhart said that about monetary policy in 1975. Fifty years later, it's the single most important sentence in AI evaluation.
The moment a benchmark gets popular, every incentive in the industry points toward optimizing for it. Not because companies are dishonest (though some are). But because benchmarks become the proxy for capability in investor decks, blog posts, and sales calls. "We score 92% on MMLU" is easier to say than "we're pretty good at the specific things your business needs."
OpenAI published a paper on measuring Goodhart's Law in reinforcement learning. The irony of the company most associated with benchmark marketing publishing research on benchmark gaming is... something.
Here's how it plays out in practice. A model lab trains a new model. They evaluate it on public benchmarks. If it doesn't score high enough, they tweak the training mix — maybe add more math data if GSM8K is low, more code if HumanEval needs help. This is rational behavior. It's also exactly the process that makes benchmarks less meaningful over time.
The ICLR 2025 paper on cheating automatic LLM benchmarks documented this formally. But practitioners already knew. We've been watching benchmark scores go up while production reliability stays flat.
Let me be precise about where benchmarks fail, because the criticism "benchmarks are useless" is as wrong as "benchmarks are truth."
What benchmarks do measure well:
What benchmarks don't measure at all:
That second list is everything that matters in production. And not a single item appears on any popular leaderboard.
One popular workaround is using one LLM to evaluate another. Sounds efficient. It's also deeply flawed.
Research throughout 2025 exposed critical issues:
I'm not saying LLM-as-judge is worthless. For rough filtering — separating obviously bad outputs from potentially good ones — it's fine. But treating it as ground truth is a mistake I've watched three teams make. Each time, they shipped a product that "passed all evals" and failed in production.
After getting burned enough times, I built my own evaluation framework. It's not glamorous. It doesn't have a leaderboard. But it predicts production issues about 10x better than any public benchmark.
This is the single most predictive metric I've found. Forget reasoning ability — can the model follow instructions?
import json
from typing import Any
def measure_format_compliance(
responses: list[str],
expected_schema: dict[str, Any]
) -> float:
"""Test if the model returns valid, parseable output."""
valid = 0
for response in responses:
try:
parsed = json.loads(response)
# Check against expected schema
if all(key in parsed for key in expected_schema):
valid += 1
except (json.JSONDecodeError, KeyError):
pass
return valid / len(responses)
I test this with 500+ prompts that request structured output. A model that returns valid JSON 99.5% of the time beats one that scores 5 points higher on MMLU but drops to 96% format compliance under load. That 3.5% gap means roughly 1 in 30 requests fails in production. At scale, that's thousands of errors per day.
Generic hallucination benchmarks are useless because hallucination is domain-specific. A model might never hallucinate about Python syntax but consistently make up medical dosages.
def measure_hallucination_rate(
questions: list[str],
ground_truth: list[str],
model_responses: list[str]
) -> dict:
"""Compare model outputs against known-correct answers."""
results = {
"total": len(questions),
"correct": 0,
"hallucinated": 0,
"refused": 0
}
for q, truth, response in zip(questions, ground_truth, model_responses):
if "I don't know" in response or "I cannot" in response:
results["refused"] += 1
elif verify_against_source(response, truth):
results["correct"] += 1
else:
results["hallucinated"] += 1
return results
I build a test set of 200-500 questions from our actual domain data, with verified correct answers. Then I measure how often the model invents facts. In my experience, hallucination rates can swing from 6% to 18% between models that score identically on public benchmarks.
The model should refuse exactly when it should — no more, no less. Over-refusal is a production problem I don't see discussed enough.
I maintain two test sets:
The target is 95%+ accuracy on both sets. Most models nail the first and fail the second.
Average latency is a vanity metric. P95 latency determines your timeout settings and user experience for the worst 5% of requests.
import numpy as np
import time
def measure_latency_distribution(
model_fn,
test_prompts: list[str],
num_runs: int = 100
) -> dict:
"""Measure latency at multiple percentiles."""
latencies = []
for prompt in test_prompts[:num_runs]:
start = time.time()
model_fn(prompt)
latencies.append(time.time() - start)
return {
"p50": np.percentile(latencies, 50),
"p90": np.percentile(latencies, 90),
"p95": np.percentile(latencies, 95),
"p99": np.percentile(latencies, 99),
"max": max(latencies),
}
A system with 200ms average latency but 5-second P99 latency will frustrate a significant number of users. I've seen models where P50 looks great but P95 is 4x higher because of occasional long-context responses. No benchmark measures this.
This one catches people off guard. I take the same prompt and rephrase it five different ways. Same intent, different wording. Then I check if the model gives consistent answers.
def measure_instruction_sensitivity(
model_fn,
prompt_variants: list[list[str]] # groups of equivalent prompts
) -> float:
"""Check if equivalent prompts produce consistent outputs."""
consistent = 0
for variants in prompt_variants:
responses = [model_fn(v) for v in variants]
# Check semantic equivalence of all responses
if all_semantically_equivalent(responses):
consistent += 1
return consistent / len(prompt_variants)
Models that score 90%+ on benchmarks sometimes give contradictory answers when you rephrase the same question. "Summarize this in 3 bullets" vs "Give me a 3-point summary" shouldn't produce wildly different outputs. But it does, more often than you'd expect.
If you're deploying an LLM and don't have an eval framework, here's where to start. This isn't theory — it's what I run before every model swap.
Step 1: Build your ground truth dataset (Week 1)
Collect 200-500 real queries from your domain. For each, write the correct answer. This is tedious. It's also the single highest-ROI investment in your ML pipeline.
Step 2: Define your metrics (Day 1)
Pick 3-5 metrics from this list. Not all of them — the ones that matter for your use case:
| Metric | What It Catches | Priority For |
|---|---|---|
| Format compliance | Broken parsers, failed integrations | Any structured output use case |
| Hallucination rate | Made-up facts, wrong answers | Knowledge-heavy applications |
| Refusal calibration | Over-blocking, under-blocking | Customer-facing products |
| P95 latency | Timeout issues, UX degradation | Real-time applications |
| Instruction sensitivity | Inconsistent behavior | Any production system |
| Cost per 1K queries | Budget overruns | High-volume applications |
Step 3: Automate and run on every model change (Week 2)
# Simplified eval pipeline
def evaluate_model(model_fn, test_suite):
results = {}
results["format_compliance"] = measure_format_compliance(
[model_fn(p) for p in test_suite["format_prompts"]],
test_suite["expected_schema"]
)
results["hallucination_rate"] = measure_hallucination_rate(
test_suite["questions"],
test_suite["ground_truth"],
[model_fn(q) for q in test_suite["questions"]]
)
results["p95_latency"] = measure_latency_distribution(
model_fn, test_suite["latency_prompts"]
)["p95"]
return results
Step 4: Set thresholds, not rankings (Day 1)
Don't pick "the best model." Pick "any model that meets these minimums":
Multiple models will meet your thresholds. Pick the cheapest one. Seriously. If two models both pass your eval suite, the one that costs less is better. Benchmarks might say Model A is "smarter," but if both solve your problem, intelligence is a waste of money.
I should be fair: not all benchmarks are equally broken.
Chatbot Arena (now LMArena) is genuinely useful. It's collected over 6 million blind pairwise votes from real users comparing 140+ models. Users never see model names. They just pick which response they prefer. The Elo ratings that come out of this process are the closest thing we have to "what do humans actually prefer."
It's not perfect — the cherry-picking scandal proved that. But the blind evaluation design means brand bias doesn't affect individual votes. That's something.
Stanford's HELM is also worth watching. It evaluates models across 42 scenarios on seven metrics including fairness, bias, toxicity, and robustness — not just accuracy. It's the most comprehensive academic benchmark available, even if it's slower to update than the leaderboard chasers want.
The practical rule: use crowdsourced benchmarks to verify that a model is generally capable. Use your own eval suite to decide what goes to production.
Most "LLM benchmark" articles fall into two camps. The optimists treat leaderboards as gospel — "Model X is best because it scores highest." The cynics dismiss all benchmarks — "numbers are meaningless, just vibe-check it."
Both are wrong.
Benchmarks are useful as rough filters. If a model scores below 60% on MMLU, it's probably not ready for production use cases that require broad knowledge. That's real signal. The problem isn't that benchmarks measure nothing — it's that they measure the wrong things for production decisions, and the industry treats them as the right things.
The other thing articles miss: the cost of being wrong is asymmetric. If you pick a model based on benchmarks and it works in production, great. If you pick it based on benchmarks and it fails — hallucinating customer data, refusing valid requests, timing out under load — the cost is enormous. Lost user trust, engineering time debugging, potential compliance issues.
Given that asymmetry, "trust but verify" isn't enough. You need "verify, then conditionally trust."
Here's something nobody talks about. There's an entire economy built around LLM benchmarks, and the incentives are misaligned.
Model labs want high scores for marketing. Benchmark creators want their benchmark to become the industry standard. Leaderboard operators want traffic. AI media wants clickbait headlines ("New Model Crushes GPT-4 on Every Benchmark!"). None of these incentives align with "help practitioners pick the right model for their specific use case."
The result is a cycle: new benchmark launches, models get optimized for it, benchmark becomes meaningless, new harder benchmark launches. Each cycle takes about 12-18 months. MMLU-Pro went from "this will separate the best models" to "saturated" in under a year.
I don't think this is fixable within the current paradigm. As long as benchmarks are public, they'll be gamed. As long as scores drive funding decisions, labs will optimize for them. The only escape is building your own evaluation — one that's specific to your use case and never published on a leaderboard.
I think the LLM benchmark ecosystem is doing more harm than good for practitioners.
Not for researchers — researchers need standardized evaluation, and benchmarks serve that purpose reasonably well. But for people building products? The benchmarks are a distraction.
I've watched teams spend weeks evaluating models on public benchmarks, building elaborate comparison spreadsheets, debating whether 87.3% or 89.1% on MMLU matters. Then they deploy the "winner" and discover it can't consistently return JSON, hallucinates their product names, and times out on 5% of requests.
The fix isn't better benchmarks. The fix is accepting that evaluation is a local problem, not a global one. Your eval suite should be as specific to your use case as your product is specific to your market. Nobody else's benchmark tests "does this model correctly classify refund requests for our specific product categories with our specific edge cases." Only you can test that.
Build a test set from your real data. Define the metrics that predict your production issues. Automate the evaluation. Run it on every model update. Treat public benchmarks as a rough pre-filter — not the final word.
The model that scores 3 points lower on MMLU but passes all your production evals at half the cost? That's the model you want. Every time.
Stop watching the leaderboard. Start watching your error logs.