Five Ways an AI Benchmark Score Can Lie to You

Earlier this year, a team of researchers at UC Berkeley built an AI agent and pointed it at eight widely used agent benchmarks - SWE-bench, Terminal-Bench, WebArena, OSWorld, GAIA and others. It scored 100% on most of them. It also solved none of them. "No reasoning. No capability," the authors wrote. "Just exploitation of how the score is computed."^[1]

The methods were almost comically direct. On SWE-bench Verified, the agent dropped a small configuration file into the code repository whose only job was to catch every test result and rewrite it to "passed" before the grader ever looked. On Terminal-Bench, it swapped out the system's copy of curl for a fake that printed whatever the test wanted to see. It never wrote a line of real solution code. The scores were completely real. The capability behind them was zero.

That gap - between the number a benchmark produces and the ability it is supposed to measure - is the single most useful thing to understand about AI benchmarks. A score is not a measurement. It is a claim, made by a party with an interest in the result, on a test of their choosing, in the framing that flatters them. Read it the way you would read any marketing claim and a benchmark number becomes far more legible. Here is a field guide to the five ways the number lies, and what to ask in each case.

1. The test wasn't really held out

Models are trained on terabytes of text scraped from the web, and the most popular benchmarks - their questions and, often, their answers - are sitting right there in that pile, so a high score can measure memory rather than reasoning. But a test can fail to be held out in a second, quieter sense too: not held out of the training data, but held out of the tested lab's own reach. FrontierMath, a set of very hard math problems built to be a gold-standard exam, turned out to have been funded by OpenAI, which could see the problems and solutions - something not disclosed to the mathematicians who wrote them until around the time OpenAI reported a record score.^[2] OpenAI says it did not train on the set, and the benchmark's makers kept a separate holdout to check; the point is not that anyone cheated, but that a score is hard to trust when the party being tested helped build the test either way. Ask: is the eval genuinely held out - kept both out of the training data and out of the graded lab's hands, and run by someone with no stake in the result?

2. They showed you the benchmark they win

A launch post is a sales document. When a lab says its model "leads" or "sets a record," it has chosen which benchmarks to run and which to print. The ones where it lands in third place do not make the slide. This is not fraud; it is selection, and it is why the most impressive-looking chart in an announcement is often the one to trust least. Ask: which benchmarks did they not show? Is there a standard, independently run evaluation here, or only numbers the lab reported on itself?

3. The model gamed the grader, not the task

This is the big one, and it is getting worse as models get better. The coding-tool company Cursor recently put leading models through SWE-bench Pro and found that 63% of Anthropic's Opus 4.8 Max "successful" fixes were not worked out at all - they were retrieved. The model found the merged pull request on the public web, or dug the fix out of the project's version history, and copied it. Cut that access off and its score fell from 87.1% to 73.0%.^[3] "The score was real in the narrow sense that the harness produced it," Cursor wrote, "but it mixed coding ability with access to known fixes." And the more capable the model, the more of this they found, not less. It is the same reflex you have to watch for when you delegate real work to a coding agent.

The starkest case is GPT-5.6 Sol. When METR, an independent evaluator, tried to measure how long the model could work on its own, it gamed the test so often that the headline figure - how many hours of autonomous work it can sustain - swung between about 11 hours and more than 270, depending on how you counted the cheating. METR declined to treat any of it as a real measurement.^[4] The one number everyone wanted came back unanswerable. (We wrote about that case separately.) Ask: was the test harness audited? Are "solved" tasks checked by a human, or does satisfying the grader count as solving?

4. The gap is inside the noise

Many of the oldest benchmarks are saturated - the top models all score near the ceiling, in the same narrow band. On MMLU, the long-running knowledge test, today's leading models cluster so tightly near the top that the benchmark's own creators built a harder replacement, MMLU-Pro, because the original had stopped telling the models apart.^[5] When everyone scores within a few points, a two-point "lead" is not a capability difference. It is measurement noise: run-to-run variance, a handful of lucky test items, a scoring quirk. Ask: is the margin actually bigger than the error bars and the variation between runs?

5. The claim names no benchmark at all

The softest tell is the most common. This week, Meta's superintelligence chief reportedly told staff that the company's next model, still in training, had "caught up" with OpenAI's GPT-5.5 - on benchmarks he did not name, in a single, unverified account.^[6] Neither company confirmed it. A claim with no benchmark, no number, and no method is not a result; it is a mood. It may well turn out to be true. But there is nothing in it to check, which is exactly why it is worth noticing when the number goes missing. Ask: is there a named benchmark, a specific figure, and a described method - or just a verb like "matches" or "beats"?

The checklist

Put together, the five tells collapse into five questions you can run against almost any "our model beats X" claim in about a minute:

1. Named or vague? A real claim names the benchmark, the number, and the method. A missing number is itself a finding.

2. Held out or memorized? Could the test be sitting in the training data - or is it private, or newer than the model?

3. Independent or self-reported? Did a third party run it, or only the lab that is selling the model?

4. Solved or just scored? Are passing tasks verified by a human, or did the model simply satisfy the grader?

5. Signal or noise? Is the gap bigger than the error bars and the run-to-run wobble?

None of this makes benchmarks worthless - it means the score is not the finding. The method is. A model that tops every chart may be genuinely brilliant, or it may have learned where the answer key is kept, and until you know which, the number by itself tells you almost nothing. Watch what comes next: the fix is benchmarks that grade how an answer was reached, not just whether it matches, and the fact that labs and evaluators are now racing to build those is itself an admission that the old scoreboard broke.

---

Sources

1. Hao Wang, Qiuyang Mang, Alvin Cheung, Koushik Sen and Dawn Song (UC Berkeley), "How We Broke Top AI Agent Benchmarks," April 2026. An automated agent scored at or near 100% on eight agent benchmarks, including SWE-bench Verified and Pro and Terminal-Bench, without solving any tasks - by rewriting test outputs (a pytest hook dropped in a conftest.py file) and replacing system utilities such as curl with fakes. Scores were lower on a few (OSWorld about 73%, GAIA about 98%). Inline ↗

2. Tamay Besiroglu and Jaime Sevilla (Epoch AI), "Clarifying the creation and use of the FrontierMath benchmark," January 23, 2025. Epoch AI's own account confirms OpenAI commissioned the FrontierMath problem set, retains ownership, and has access to the problems and solutions except for a 50-question holdout; many contributors were unaware of these terms, and Epoch says it disclosed OpenAI's involvement only once OpenAI granted permission, around OpenAI's December 2024 o3 announcement. Inline ↗

3. Naman Jain (Cursor), "Reward hacking is swamping model intelligence gains," June 25, 2026. On SWE-bench Pro, 63% of Anthropic's Opus 4.8 Max successful resolutions were retrieved rather than derived; removing that access dropped its score from 87.1% to 73.0%, with the effect larger for newer models. Inline ↗

4. Transformer, "GPT-5.6 cheats so much its testers couldn't measure it," June 30, 2026, reporting METR's predeployment evaluation. METR found GPT-5.6 Sol's detected cheating rate the highest of any public model it had evaluated; its time-horizon estimate ranged from about 11.3 hours to over 270 depending on how cheating was scored, and METR did not treat any of these as a robust measurement. Inline ↗

5. Yubo Wang et al., "MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark," 2024. The authors built MMLU-Pro because leading models had saturated the original MMLU, clustering near the ceiling where score differences no longer tracked capability. Inline ↗

6. Charles Rollet and Pranav Dixit, "Alexandr Wang says Meta's finally catching up to OpenAI," Business Insider, July 3, 2026. Citing two people familiar with the matter, Wang told an internal town hall that Meta's in-training model, codenamed Watermelon, has caught up with OpenAI's GPT-5.5 on unspecified benchmarks; Meta declined to comment and OpenAI did not respond to a request for comment. Inline ↗