- Hugging Face’s ICML 2026 Agent Reproductions Challenge deployed AI coding agents to verify specific claims in top research papers, running July 15 to August 2.
- AI coding agents caught a bug in a published sampler that had survived traditional peer review, surfaced through automated codebase execution logged via Hugging Face’s Trackio library.
- The challenge awarded prizes for “Best Falsification / Negative Result,” directly rewarding agents for finding what does not replicate rather than confirming what does.
A bug in a published sampler survived peer review but not an AI coding agent. Hugging Face‘s ICML 2026 Agent Reproductions Challenge, which ran July 15 to August 2, sent agents built on Claude, Cursor and OpenCode into published codebases to audit specific claims in top ICML 2026 papers, logging every step in a public verification trail now frozen ahead of winner announcements.
Bugs the Reviewers Missed
Human researchers can spend weeks wrestling with unmaintained repositories before confirming a result will not replicate. The Agent Reproductions Challenge cut that loop by sending coding agents directly into published codebases: navigating dependencies, recreating environments and logging every step via Hugging Face’s Trackio library. Trackio captures code simplifications, package installation failures and experimental outputs into a public logbook. The sampler bug surfaced during this process is the clearest example of what systematic automated execution finds that static review does not.
Broken Proofs, Not Just Broken Code
The challenge went beyond running scripts. Agents also targeted theoretical claims, with at least one reproduction aimed at a paper titled “BrokenMath: Sycophancy in Theorem Proving,” probing whether the mathematical arguments hold under automated re-examination, not just whether the accompanying code runs. A model that assists in theorem proving is only as reliable as the theorems it was trained on.
Data and Benchmark Integrity
Reproducibility failures in AI research often live upstream of the model: in data splits, preprocessing steps and evaluation scripts that rarely get scrutinised after publication. An OECD report on AI in science cites a high irreproducibility rate across AI research, with problems spanning image recognition, natural language processing and reinforcement learning. The challenge’s claim-by-claim structure forced agents to engage with those specifics. When an agent fails to replicate a reported performance figure, that failure typically points somewhere in the data pipeline, not the model architecture.
How the Auditing System Works
The challenge organisers have described the approach as “Autonomous Scientific Auditing.” Every agent action is captured into an auditable artifact, creating a public logbook that persists independent of the contest format.
For builders thinking about using AI agents for research and information gathering the Trackio-plus-logbook pattern is worth examining on its own merits. Capturing every agent action into an auditable artifact is good engineering practice regardless of whether a judge ever reads it, and it translates cleanly outside the competition context. Teams running agents against fragmented data pipelines face the same traceability problem.
Incentives Matter
The most interesting structural choice here was awarding prizes for “Best Falsification / Negative Result.” That directly flips the usual incentive: rather than rewarding confirmation, it pays for discovering what does not replicate. Academic publishing has struggled with this for decades. Coding agents, which have no stake in a paper’s conclusions, are a natural fit for the job. They will follow a failing execution path as diligently as a successful one.
The verdicts are now frozen. Winners are yet to be announced.
