AI Benchmarks Are Way Less Reliable Than You Think

We All Do the Same Thing

When you're trying to pick an AI model, you look at the benchmarks. GPT-5 beats Gemini on MMLU, Claude dominates reasoning, Llama surprises on code. These leaderboards guide decisions for millions of users and thousands of companies. The problem? Those scores rest on shakier ground than anyone wants to admit.

A Google Research study, presented at AAAI 2026, puts a finger on a fundamental flaw: the humans evaluating AI responses don't agree with each other. And most benchmarks pretend that disagreement doesn't exist.

The Consensus That Doesn't Exist

Building a benchmark requires reference data. Someone has to decide whether a model's response is "good" or "bad," "toxic" or "acceptable." That someone is a human annotator. Typically, you get between 1 and 5 per question.

Here's the thing: when you ask multiple people whether a comment is toxic, whether a response is helpful, or whether a text is offensive, opinions diverge. That's normal - we're talking about subjective judgments, not mathematics. But benchmarks treat those judgments as objective truths.

The Google study, titled "Forest vs Tree," tested thousands of combinations across four different datasets, including one with 107,620 comments rated by 17,280 annotators. Their conclusion: with fewer than 10 evaluators per item, results aren't reliably reproducible. The standard practice of 1 to 5 annotators? Wildly insufficient.

Forest or Tree: A Choice That Changes Everything

Google's researchers framed the problem as a trade-off between two approaches. With a fixed annotation budget, you can either evaluate lots of items with few people each (the forest), or evaluate fewer items with more evaluators (the tree).

The right answer depends on what you're measuring. If you just want to know which model is "overall better" by majority vote, the forest works. But if you want to capture the diversity of human opinions, you need the tree - more evaluators per question.

The good news: by optimizing this ratio, you can get reliable results with just 1,000 total annotations. But a bad ratio makes conclusions fragile, even with a bigger budget.

A Problem Bigger Than Disagreement

Annotator disagreement is just one piece of the puzzle. AI benchmarks are facing a credibility crisis on multiple fronts.

Gaming the leaderboards. A study by Cohere, Stanford, MIT, and the Allen Institute revealed that Meta tested 27 private variants of Llama 4 on the LMArena platform before launch, publishing only the best score. The version submitted to the leaderboard was "optimized for conversationality" and differed from the version actually made available to the public.

Data contamination. Models like GPT-5.2, Claude Opus 4.5, and Gemini 3 Flash were able to reproduce benchmark code word for word, suggesting they'd been exposed to these tests during training. It's like taking an exam after getting access to the answer key.

Test saturation. MMLU scores now exceed 90%. Stanford's HAI 2025 report shows staggering jumps: GPQA surged 48.9 points in a year, SWE-bench 67.3 points. When everyone scores 90%, the grade stops meaning much.

The Gap Between Score and Reality

The number that says it all: on SWE-bench Verified, top models hit around 80% success. On SWE-Lancer, a benchmark based on real freelance projects, that score drops to 26.2%. The gap is massive.

Another telling example: researchers discovered that an X-ray classifier, supposedly detecting collapsed lungs, had actually learned to spot chest drains visible in the images. When images with drains were removed, performance dropped 20%. The model had great scores - for all the wrong reasons.

This is the "construct validity" problem: benchmarks don't always measure what they claim to measure.

What to Do While Waiting for Better Benchmarks

The Google study offers concrete tools. The researchers open-sourced a simulator that lets you test different N/K ratios before building a benchmark. It's a first step toward more rigorous evaluations.

Other initiatives are emerging: SWE-bench Pro with stricter criteria (scores drop to 23%), dynamic benchmarks that change regularly to prevent contamination, and domain-specific evaluations rather than generalist leaderboards.

But until the ecosystem catches up, the takeaway is simple: never trust a single benchmark. Cross-reference sources, test models on your own use cases, and keep in mind that behind every score, there are humans who probably couldn't agree with each other.

Benchmarks aren't useless. But they're far less objective than we think. And knowing that is exactly how you use them best.