SWE-bench Pro Says Claude Fable 5 Beats GPT-5.6 Sol by 16 Points

As of mid-July 2026, the two leading AI coding systems aren’t actually racing on the same benchmark anymore. OpenAI’s GPT-5.6 Sol and Anthropic’s Claude Mythos 5/Fable 5 both post strong numbers on SWE-bench Verified, but that benchmark is now widely regarded as compromised by training contamination. On SWE-bench Pro, the benchmark that’s replaced it, Mythos 5/Fable 5 leads GPT-5.6 Sol by 16 points: 80.3% versus 64.6%. The question worth asking isn’t which number is bigger, it’s which benchmark you should trust.

Why SWE-bench Verified No Longer Settles the Question

SWE-bench Verified tests whether a model can autonomously resolve real GitHub issues in popular open-source Python repositories. That is a harder problem than generating a function from a prompt: it requires reading a codebase, identifying a fault and producing a working patch.

That framing matters less than it used to. By early 2026, SWE-bench Verified had accumulated enough training-data overlap that OpenAI reportedly stepped back from treating it as a reliable differentiator. SWE-bench Pro was built specifically to fix this: it draws from actively maintained repositories rather than a fixed set, so the test set can’t simply be memorized.

Details on the Mini-SWE-agent are limited in the benchmark context, but the general pattern is well-established. An agentic scaffold gives the model access to a terminal, the ability to run tests and read error output, and a structured loop for iteration. Without those affordances, even a strong model works from a static snapshot of the problem. With them, it can do something closer to actual debugging.

Anthropic’s Claude models maintain strong contention

Mythos 5 and Fable 5 are the same underlying model. Fable applies additional safety classifiers to sensitive cyber and biological requests, routing some of them to Claude Opus 4.8 instead. That distinction matters more than the benchmark number: Mythos 5 remains restricted to a small number of vetted partner organizations, while Fable 5 is broadly available. For most teams evaluating these models, Fable 5, not Mythos 5, is the relevant comparison point, since it’s the version they can actually access.

The rising impact of agentic frameworks

The gap between GPT-5.6 Sol’s SWE-bench Verified and Pro scores points to something structural about how these evaluations work. A raw language model, however capable, receives a problem description and produces output in one pass. An agentic system treats the same problem as a loop: read the codebase, form a hypothesis, apply a change, run the tests, read the failure output, revise. SWE-bench Verified rewards the loop. That is a reasonable approximation of how real software bugs get fixed, not a flaw in the benchmark.

The practical implication for development teams is that the choice of agent framework may matter as much as the choice of model. For teams already thinking through those trade-offs, our coverage of LangGraph’s debugging features covers some of the relevant tooling decisions in detail.

Beyond SWE-bench: diverse coding benchmarks

DeepSeek-Coder-V2-Instruct scored 90.2 on HumanEval and 76.2 on MBPP+, outperforming earlier closed-source models including GPT-4 Turbo, Claude 3 Opus and Gemini 1.5 Pro across several coding and mathematical benchmarks. DeepSeek’s models are open-source and trained across more than 80 programming languages, which affects both their strengths and the conditions under which those benchmark scores were produced.

Google’s Gemini 1.5 Pro, noted for long-context reasoning, scores below those leaders on some aggregate coding metrics, with its 002 update showing improved HumanEval performance. Gemini 3.5 Flash, according to Google, is optimised for coding and parallel agentic execution, though independent benchmarking of those specific claims remains limited.

On Artificial Analysis’s own Coding Agent Index, GPT-5.6 Sol (paired with OpenAI’s Codex harness) leads at 80 points, while trailing Claude Fable 5 by one point on their broader Intelligence Index, at roughly a third of the cost. The pattern holds across evaluators: which system wins depends heavily on which harness and which benchmark you’re using.

The nuance of benchmark scores in real-world impact

Benchmark contamination is a genuine problem. If a model was trained on data that included the test problems, its score reflects memorisation as much as capability. SWE-rebench attempts to address this by continuously pulling fresh GitHub issues and tracking problem creation dates against model release dates. The gap that methodology reveals can be significant: a model that scores well on a fixed benchmark may solve far fewer genuinely unseen problems, though the degree varies by model and is not uniformly documented.

SWE-bench Verified’s decline is itself evidence of how quickly this problem compounds: a benchmark can be state of the art one year and unreliable the next as models train on overlapping data. SWE-bench Pro’s rolling refresh is an attempt to stay ahead of that cycle, but it will face the same pressure eventually.

Leaderboard positions are directional indicators, not performance contracts. The benchmarks above test specific, well-defined problem types under controlled conditions. Real codebases are messier: unfamiliar dependencies, incomplete specifications and requirements that shift mid-task. Contextual comprehension, multi-language support and integration with existing developer tooling all affect whether benchmark performance translates into working hours saved. For most teams evaluating these systems, an 80.3% versus 64.6% gap on SWE-bench Pro still matters far less than how each performs on their own codebase. For more coverage of AI research and breakthroughs, visit our AI Research section.