The Benchmark Says 96 Percent. Your Repo Gets 47.

Last updated: August 2026

On February 23, 2026, OpenAI published a research post explaining why it had stopped reporting SWE-bench Verified scores, and asked the rest of the industry to do the same. Five months later, on July 24, 2026, Anthropic's Claude Opus 5 system card reported 96.0 percent on that same benchmark.

Neither party is lying. That is exactly why the mechanism is worth understanding. A benchmark number is a measurement of a model, a scaffold, a grading harness, and a dataset, all at once. Change any one of those four and the number moves by tens of points without the underlying capability moving at all. When a vendor quotes 96 and your pilot lands somewhere near 50, nobody has been deceived. You were reading an answer to a question you did not ask.

The first half of this piece is a ladder for one capability, agentic software engineering, measured five ways inside three weeks. The second half is the method for measuring it yourself, most of which a frontier lab already wrote down in the methodology sections of its own system card, where nobody reads it.

Two Labs, One Benchmark, Two Defensible Positions

OpenAI's audit is specific. Engineers reviewed 138 SWE-bench Verified problems that OpenAI o3 failed to solve consistently across 64 independent runs, roughly 27.6 percent of the 500-problem set. Each case was reviewed independently by at least six experienced software engineers. The verdict: 59.4 percent of those 138 problems had material issues in test design or problem description, which OpenAI describes as rendering them extremely difficult or impossible even for the most capable model or human.

• 35.5 percent were narrow. Tests enforce a specific implementation detail. The canonical example is pylint-dev__pylint-4551, where the hidden tests import a function called get_annotation that the problem statement never mentions. Correct fixes fail on an import error.
• 18.8 percent were wide. Tests check behavior the description never asked for. In sympy__sympy-18199, the source pull request closed three separate nthroot_mod issues and the task text describes only the last of them. Models implement the described fix and fail the tests covering the other two.
• The remaining 5.1 percent is arithmetic, not a category. OpenAI names narrow and wide; 59.4 minus 35.5 minus 18.8 is what is left over. Worth knowing before you quote it as a third finding.

Then contamination. OpenAI ran an automated red-teaming setup in which GPT-5 probed GPT-5.2, Claude Opus 4.5, and Gemini 3 Flash for memorized content. Every frontier model tested reproduced either the human-written gold patch or verbatim problem-statement specifics on at least some tasks. The post also notes that state of the art on Verified moved from 74.9 to 80.9 percent over six months, which is the shape a benchmark makes on its way to saturation.

Anthropic's 96.0 is equally specific: an average over five trials, adaptive thinking at max effort, on the standard 500-problem set. Both numbers are honest. One is a statement about a dataset that no longer separates frontier models, and the other is a statement about the same dataset. That is the whole disagreement.

The Ladder: One Capability, Five Ways to Measure It

Here is the same skill measured five ways, all figures read on the dates shown. This table does more work than any model comparison chart, because every row is the same capability.

The bottom two rows are the ones your project will feel. Scale's private split is 276 instances sourced from 18 private, proprietary codebases from startups, acquired through partnerships and not publicly accessible. No model in that table could have trained on them. That is what code you actually own looks like to an agent.

Where the Drop Actually Comes From, and Where It Does Not

The tempting story is that vendor scaffolds inflate numbers and independent harnesses deflate them. Check it against row two. Vals AI, running its own harness, puts Claude Opus 5 at 97.00 percent on SWE-bench Verified, a point above the vendor's own figure, on a benchmark where 3 of 79 evaluated models now clear 95 percent. Independent replication does not pull Verified down, because the scaffold was never the problem there. The dataset was.

The drop has three causes. Two of them are tangled together and the third is not, which matters if you plan to reason from these numbers:

• Contamination and saturation. Verified's 500 problems and their fix commits are public, and OpenAI showed frontier models can recite parts of both. A score on a dataset a model has partly memorised is a recall measurement wearing a capability costume.
• Task shape. SWE-bench Pro is 1,865 problems from 41 actively maintained repositories, deliberately long-horizon and multi-file rather than the single-function patches that dominate Verified. Rows one and three change both variables at once, same model and week and vendor and scaffold, 96.0 on Verified and 79.2 on Pro. Those two numbers alone cannot tell you how the 17 points split between leakage and difficulty, and anyone who says it is all leakage is guessing.
• Model recency explains more of the last two rows than most people admit. Scale's leaderboards run every model through one standardized harness, and that discipline costs time. As of August 11, 2026 the newest Claude entry is Opus 4.6 and the newest OpenAI entry is GPT-5.4, while vendors are publishing Opus 5 and GPT-5.6 Sol. Independent boards lag the frontier by roughly a generation.

One disclosure the leaderboard does not carry: Meta holds a 49 percent non-voting stake in Scale AI from a 14.3 billion dollar investment in June 2025, and Meta's Muse Spark 1.1 currently tops both the public and private splits. I am not alleging anything. I am saying that if your decision rests on a three-point gap at the top of somebody else's leaderboard, you have built your decision on the wrong foundation regardless of who owns whom.

Two Things No Leaderboard Will Ever Surface

Refusal accounting can move a headline score by tens of points. Vals AI states it plainly on its GPQA Diamond page: the published 93.18 percent for Claude Fable 5 counts refusal-triggered fallbacks as successes, and when refusals are counted as failures the score falls to 55.56 percent, at a 41.92 percent refusal rate (read August 9, 2026). That is a 37-point swing from a scoring convention, not from a capability.

The vendor's own document shows the same problem from a different angle. In the Opus 5 system card's FrontierBench section, Anthropic reports that Fable's safety classifiers flagged 42 percent of API calls on 26 percent of trials, falling back to Opus 4.8, while Opus 5's classifiers flagged and refused 5 percent of calls on 4 percent of trials. Resist the urge to read those two 42s as one finding. One is a share of questions on a closed-book science benchmark, the other a share of API calls in a terminal-agent run, and the match is a coincidence. What survives the comparison is the shape: an eightfold spread in refusal rate between two models from the same vendor, and a scoring convention that can bury it. If your workload sits near security tooling or life sciences, that is the number that decides your rollout, and it appears on no leaderboard.

A model can get more accurate and more hallucinatory in the same release. On the public split of AA-Omniscience, a closed-book factuality benchmark, the system card reports that Claude Opus 5's accuracy is 11 percent higher than Opus 4.8 while its rate of hallucinations is also 6 percent higher. Those are not contradictory. Accuracy and abstention are separate dials, and a single composite score hides which one moved. If you are putting a model in front of customers, the second number matters more than the first.

Step 1: Build a Golden Set the Model Could Not Have Seen

Everything above collapses into one requirement: your tasks have to postdate the model's training cutoff and live somewhere it could not crawl. Anthropic applies this to itself in the same card, evaluating on the June 2026 release of ArxivMath specifically to avoid contamination with Opus's training data, and reporting DeepSWE as 113 tasks written from scratch to avoid benchmark contamination.

You already have the raw material. Your merge history is a stream of dated, verified, multi-file changes with reference solutions attached. Thirty to fifty tasks is enough to be useful, and the confidence interval math in the last section will tell you when it is not.

#!/usr/bin/env python3
"""Build eval tasks from merges that landed after a model's knowledge cutoff.

The whole point is the date filter. A task merged after the cutoff cannot be in
the weights, so a score on it is a capability measurement rather than a recall
measurement. Run this against a private repo, never a public one.
"""
import json
import subprocess

REPO = "/srv/checkouts/billing-api"
CUTOFF = "2026-05-01"        # the model's stated training cutoff
MIN_FILES = 2                # a one-line diff is not a capability test


def sh(*args):
    return subprocess.run(
        args, cwd=REPO, capture_output=True, text=True, check=True
    ).stdout


def merges_since(cutoff):
    log = sh("git", "log", "--first-parent", "--merges", "--no-decorate",
             "--since=" + cutoff, "--pretty=%H%x1f%cI%x1f%s")
    for line in log.strip().splitlines():
        yield line.split("\x1f")


tasks = []
for sha, merged_at, subject in merges_since(CUTOFF):
    base = sh("git", "rev-parse", sha + "^1").strip()
    # NOTE: git show --name-only prints nothing for a merge commit.
    # Diff the two commits instead.
    files = sh("git", "diff", "--name-only", base, sha).split()
    if len(files) < MIN_FILES:
        continue
    tasks.append({
        "task_id": sha[:12],
        "merged_at": merged_at,
        "base_commit": base,                       # where the agent starts
        "issue_text": subject,                     # swap in the linked issue body
        "reference_patch": sh("git", "diff", base, sha),
        "files_touched": files,
    })

print(len(tasks), "candidate tasks merged after", CUTOFF)
with open("golden_set_raw.json", "w") as fh:
    json.dump(tasks, fh, indent=2)

One warning that will cost you a day if you skip it: git show --name-only returns nothing for a merge commit. Diff the merge against its first parent instead, which is what the script above does.

Step 2: Check Your Reference Solutions Before You Trust Any Score

This is the step nobody does, and it is the one that invalidates the most work. If a task's own reference solution fails the task's own tests, then every model score on that task is noise, and you will spend a week tuning prompts against a broken grader.

The precedent is in the system card. On ProgramBench, a 200-task long-context coding benchmark, Anthropic writes that it excluded 34 tasks for which the reference binary itself scores below 0.9 on the hidden test suite, indicating test flakiness, leaving 166 tasks, and within those scores only against tests the reference binary passes. That is 17 percent of the benchmark discarded because the known-correct answer could not pass. Assume your own set is at least that bad on the first pass.

#!/usr/bin/env bash
# Validate the golden set against itself BEFORE any model sees it.
# Every task's own reference patch has to pass the task's own suite, twice,
# or the task is measuring flakiness rather than capability.
set -uo pipefail
mkdir -p quarantine

keep=0; drop=0
for task in golden_set/*/; do
  id=$(basename "$task")
  patch="$PWD/$task/reference.patch"
  git -C repo checkout -q "$(cat "$task/base_commit")"

  if ! git -C repo apply --index "$patch" 2>/dev/null; then
    echo "DROP $id  reference patch does not apply"
    mv "$task" "quarantine/$id"; drop=$((drop+1)); git -C repo reset -q --hard; continue
  fi

  ok=1
  for run in 1 2; do                      # run twice: one green run proves nothing
    timeout 900 make -C repo test > "$task/reference_run_$run.log" 2>&1 || ok=0
  done
  git -C repo reset -q --hard

  if [ "$ok" -eq 1 ]; then
    echo "KEEP $id"; keep=$((keep+1))
  else
    echo "DROP $id  reference fails its own suite"
    mv "$task" "quarantine/$id"; drop=$((drop+1))
  fi
done

total=$((keep+drop))
echo "kept=$keep dropped=$drop discard_rate=$(( total > 0 ? 100*drop/total : 0 ))%"

Run the suite twice, not once. One green run proves nothing about a flaky test, and flaky tests are how a model that did the work correctly ends up scored as a failure.

Step 3: Judge With a Panel, and Require Unanimity

For anything you cannot grade with a test runner, you need a model judge, and a single judge from the same family as the candidate is close to worthless. It shares the house style, the formatting habits, and a good share of the blind spots.

The structure worth copying is the one Anthropic used for IMO 2026. A model wrote a rubric for each problem, checked against published reference solutions. Each of Claude Opus 5's solutions was then judged independently against that rubric by a panel of three frontier models, and an answer counted as correct only when all three judges agreed. Anthropic then had human experts grade one pre-specified solution per problem. Panel and humans agreed on all six, 7 out of 7 each, for a 42 out of 42 gold.

Copy the structure, not the roster. Anthropic's three judges were Gemini 3.1 Pro, Claude Opus 4.6, and Claude Mythos Preview, so two of the three shared a vendor with the candidate. On a competition proof, where grading is close to verification against a rubric, that is defensible. On the fuzzier artifacts you will actually be grading, a tone rubric, an ops runbook, a customer-facing reply, it is precisely the correlated-blind-spot failure you set up a panel to avoid. Use three vendors.

"""Grade one submission with a three-family panel. A pass needs all three."""
import json
import anthropic

RUBRIC = open("rubric.md").read()
client = anthropic.Anthropic()

SCHEMA = {
    "type": "object",
    "properties": {
        "verdict": {"type": "string", "enum": ["pass", "fail"]},
        "criterion_failed": {"type": "string"},
    },
    "required": ["verdict", "criterion_failed"],
    "additionalProperties": False,
}


def judge_claude(task, submission):
    r = client.messages.create(
        model="claude-opus-5",
        # max_tokens caps thinking AND response text together on Opus 5.
        # The verdict is 30 tokens; the reasoning that produces it is not.
        # Size this too tight and you get stop_reason="max_tokens" and no JSON.
        max_tokens=8000,
        thinking={"type": "adaptive"},
        output_config={"effort": "high",
                       "format": {"type": "json_schema", "schema": SCHEMA}},
        system="Grade only against this rubric. Nothing outside it counts.\n\n" + RUBRIC,
        messages=[{"role": "user",
                   "content": "TASK\n" + task + "\n\nSUBMISSION\n" + submission}],
    )
    text = next(b.text for b in r.content if b.type == "text")
    return json.loads(text)["verdict"]


# judge_gpt and judge_openweight are the same function shape against their own
# SDKs. Different families, so they do not share a house style with the
# candidate or with each other.
PANEL = [judge_claude, judge_gpt, judge_openweight]


def grade(task, submission):
    votes = [j(task, submission) for j in PANEL]
    if all(v == "pass" for v in votes):
        return "pass"
    if all(v == "fail" for v in votes):
        return "fail"
    return "human_review"   # a split panel means your rubric is ambiguous

The human_review branch is the valuable part. A split panel almost never means the models are confused. It means your rubric has a criterion two readers can interpret differently, and you should fix the rubric before you fix anything else. Also write the rubric first, against a known-good artifact, and check it against a reference solution before it grades a single candidate.

Step 4: Block the Retrieval Paths, Then Re-Grade the Transcripts

The moment your eval gives an agent web access, contamination stops being a training-data problem and becomes a runtime problem. The agent will find the answer, and the answer will look like reasoning in the transcript.

Anthropic's Humanity's Last Exam methodology is a two-layer defense worth stealing. Layer one is a URL blocklist applied to both the searcher and the fetcher, published in full in the system card appendix: substring matching on normalized URLs, covering huggingface.co, lastexam.ai, the relevant arXiv IDs, and the study-help sites where solutions get posted. Layer two is the one people skip. Anthropic used Claude Opus 4.8 to review every transcript and flag any that appear to have retrieved answers from benchmark-specific sources, and confirmed cases were re-graded as incorrect. BrowseComp runs got the same treatment: scanned after the fact by a verifier, and any problem flagged for answer leakage counted as incorrect.

For an internal eval this is cheap. Your blocklist is your own wiki, your ticket tracker, and any repo where the reference patch is visible. Then run a cheap model over the transcripts asking one question: did this agent read the answer, or derive it? Re-grade the hits as failures.

Step 5: Sweep Reasoning Effort, Do Not Pin It to Max

More thinking is better is the intuition, and it is wrong often enough that pinning your eval to max effort will mislead you about both quality and cost. The clearest counterexample is in the same system card, on a benchmark run and scored by a third party.

FrontierCode is 150 agentic coding tasks built by Cognition from real pull requests, run and scored by Cognition rather than by Anthropic. Across the five reasoning-effort settings, Claude Opus 5 reaches its best main-set score of 53.4 at medium, and its best extended-set score of 63.6 at medium. Not xhigh. Not max. All figures as published July 24, 2026.

FrontierBench shows the other half of the curve, and max is not the peak there either. Anthropic puts max at 43 percent against xhigh's 44.4, close enough that the card calls it within noise. That is the whole argument in one line: you paid for the extra thinking and bought nothing measurable. Below the peak the tradeoff is real and legible. Dropping from xhigh to high costs 5 points of mean reward and saves 19 percent of output tokens; dropping to low costs about 19 points and saves 64 percent. Sweep every level on your own set and plot score against billed cost per task. On plenty of routine workloads the knee is at medium.

Step 6: Rank by Confidence Interval Overlap

Almost every internal eval writeup I have been handed reports a single percentage per model with no interval, and then recommends a migration on the strength of a one-point win. On a 200-task set, a one-point win is nothing. It is not even a weak signal.

Scale's SEAL boards handle this in one line of legend, and it is the single most copyable convention on the page: Rank (UB): 1 + the number of models whose lower CI bound exceeds this model's upper CI bound. That is why the public split shows two models tied at rank 1 despite 61.50 and 59.10 point scores, and why the private split intervals run to +/- 6.07 on 276 tasks. Report the interval and most of your leaderboard collapses into ties, which is the honest picture.

"""Rank models by confidence interval overlap, not by point score."""
import numpy as np

rng = np.random.default_rng(0)


def bootstrap_ci(outcomes, n_boot=10000, alpha=0.05):
    """outcomes: list of 1 (solved) / 0 (not solved), one per task."""
    x = np.asarray(outcomes, dtype=float)
    draws = rng.choice(x, size=(n_boot, x.size), replace=True).mean(axis=1)
    lo, hi = np.percentile(draws, [100 * alpha / 2, 100 * (1 - alpha / 2)])
    return x.mean(), lo, hi


def rank_ub(scores):
    """Scale's SEAL legend, implemented: rank is 1 + the number of models
    whose lower CI bound exceeds this model's upper CI bound."""
    return {
        name: 1 + sum(1 for other, (_, lo_o, _) in scores.items()
                      if other != name and lo_o > hi)
        for name, (_, _, hi) in scores.items()
    }


runs = {
    "model_a": [1] * 118 + [0] * 82,   # 59.0% of 200 tasks
    "model_b": [1] * 112 + [0] * 88,   # 56.0% of the same 200
}
scores = {k: bootstrap_ci(v) for k, v in runs.items()}
for name, (mean, lo, hi) in scores.items():
    print("%-8s %6.1f%%  95%% CI [%.1f%%, %.1f%%]" %
          (name, 100 * mean, 100 * lo, 100 * hi))
print(rank_ub(scores))   # both rank 1: at n=200 a 3 point gap is noise

Run that and you will see two models with a 3-point gap both land at rank 1. If your set is small enough that everything ties, the conclusion is not that the models are equal. It is that you need more tasks, or a harder split, before you are allowed to have an opinion.

What This Costs, and Where I Would Cut

A first pass costs about a week of one engineer: two days extracting and pruning tasks, one on the reference check and the quarantine pile, one on the judge rubric, one wiring up the effort sweep and interval reporting. After that it is a CI job you re-run on every model release, and the marginal cost is API spend.

If you only have two days, do steps 1, 2 and 6. A dated golden set, a validated reference pass, and a confidence interval will already tell you more than any leaderboard, because they are measuring the thing you are actually buying. The judge panel and the effort sweep are refinements on top of a measurement that is at least pointed in the right direction.

And keep reading the public numbers. They are genuinely useful for tracking where the frontier is and for knowing which model to shortlist. Our enterprise model comparison is the shortlisting step, not the deciding step. Just stop treating a headline percentage as a forecast of what happens inside your repository. It never was one, and the labs publishing those numbers now say so out loud.