ICML 2026 Tightens Rules as LLM Policy Shows Near-Zero Effect on Scores

AI Conferences Compare Quality Control: ICML Rules, NeurIPS Types, ICLR Open Review
Key Takeaways

  • An arXiv study of ICML 2026’s LLM review policies found score differences of just 0.002 points across 24,661 submissions, suggesting reviewers either ignored the rules or that permitted LLM assistance changed little about actual evaluation.
  • NeurIPS 2026’s “Contribution Types” framework formally legitimises negative results and infrastructure work by requiring reviewers to judge novelty relative to the author’s selected contribution category, rather than against a single top-down threshold.
  • ICLR’s fully public review model, including rejected and withdrawn papers deanonymised after notification, changes reviewer incentives in ways ICML’s enforcement approach cannot replicate, though the evidence that open review improves accepted-paper quality remains contested.

A randomised study posted to arXiv on September 16, 2026 tested whether ICML 2026’s LLM review policies actually changed reviewer behaviour, and the answer was essentially no: scores differed by 0.002 points across nearly 25,000 submissions. That finding lands in the middle of a genuine methodological split among top AI conferences, each betting on a different mechanism to keep research quality high as submission volumes grow.

ICML Tightens the Rules

ICML 2026 announced its policy changes in January 2026, targeting what it called “thinly sliced contributions”: papers that make minimal advances on a theme, effectively splitting one result across several submissions. The new concurrent submission rule treats all papers by the same author on a related topic as prior work, requiring cross-citation and explicit discussion. Violations can trigger desk rejection of every offending submission, not just one.

The LLM rules follow a similar logic. Authors bear full responsibility for their content; LLMs cannot be listed as authors; prompt injection leads to direct rejection. For reviewers, the policy permits LLMs to help with comprehension or prose polish, but bars them from judging a paper or drafting review text. The rationale is that review quality is a human responsibility the conference cannot outsource.

The arXiv study put that to the test by randomising 17,886 reviewers into a strict-ban group and a limited-assistance group. Score differences were negligible. The authors suggest reviewers may simply be ignoring the rules, or that LLM assistance at the permitted level does not change evaluation much either way. Neither reading is comfortable for an enforcement-first model.

NeurIPS Redefines What Counts

Rather than tightening submission rules, NeurIPS 2026 redefines novelty itself, sorting submissions into labelled “Contribution Types” and guiding reviewers to interpret originality criteria relative to the category the authors selected.

This formally legitimises a broader range of work, including negative results and infrastructure contributions, that single-threshold novelty standards tend to screen out. NeurIPS also discourages thin-slicing under its dual submission policy, so it shares ICML’s concern; it addresses it through taxonomy rather than enforcement. The conference uses OpenReview for process management while maintaining double-blind review at the reviewer and area chair levels, drawing on the platform’s tooling without committing to fully public deliberation.

The Case for Open Review

ICLR has run public peer review through OpenReview for years, and the 2026 cycle continues that approach: submissions, reviews, meta-reviews, rebuttals and author-reviewer discussions are all made public, including for rejected or withdrawn papers, which are deanonymised after the notification date.

The transparency argument is that multiple public perspectives catch incremental work and reviewer bias more reliably than any internal ruleset can. A reviewer who knows their comments will be public has different incentives than one writing for a closed committee. The drawback is cultural: researchers have to accept that a rejected submission, and the reviewers’ reasoning, will be permanently visible. That exposure changes how authors write papers and how reviewers write reviews, and not always in productive directions.

Three Bets, One Problem

ICML, NeurIPS and ICLR are running three distinct quality-control strategies simultaneously, which makes the 2026 cycle an unusually clear comparison. ICML is betting on top-down rule enforcement; the arXiv study suggests that bet has limits. NeurIPS is betting on redefined criteria; it is too early to measure the effect on accepted-paper quality. ICLR is betting on community scrutiny; the evidence base for open review improving outcomes is older and more accumulated, though still contested.

For researchers and the teams that track AI literature, the practical consequence is that the same piece of work could be evaluated very differently depending on which conference it targets, and not only because of different subject-matter fits. The review philosophy itself is now a variable. Reproducibility failures surfaced through agentic review tools at ICML 2026 suggest the quality problem is broader than any single policy can address.

Where This Leaves the Field

The 0.002-point score difference is the sharpest data point the field currently has on LLM review policy effectiveness, and it is not encouraging for enforcement advocates. A hybrid model combining structured contribution categories, along the lines of NeurIPS’s taxonomy, with post-decision transparency in the ICLR style has intuitive appeal. No major conference has fully committed to that combination, and the ICML study suggests that even well-designed policies need monitoring mechanisms the field has not yet built.

Taylor Voss
Taylor Voss

Taylor Voss is a research correspondent covering AI breakthroughs, model releases, and the science shaping the future of artificial intelligence. Taylor translates complex research into clear, compelling stories for curious readers who want more than the headlines.

📰 Journalists welcome — cite Auton AI News with attribution. Press & Media → | press@autonainews.com