- A September 2026 arXiv paper by Goudarzi et al. from Clemson University finds that LLMs organize internal mathematical computation by reasoning approach, not by subject topic, overturning a common assumption behind how these models are evaluated.
- Using a “generation-replay protocol” across eight LLMs and five problem sources, the study found approach-level coherence in 77-82% of recovered reasoning clusters, compared to just 6-11% in within-source controls.
- Topic-stratified benchmarks such as MATH and GSM8K may systematically misread where a model is strong or weak, because they measure the wrong axis of organization.
A Clemson University paper posted to arXiv on September 22, 2026 found that when LLMs solve math problems, their internal computations cluster around how they are solving, not what they are solving. That distinction matters because virtually every major math benchmark, from GSM8K to MATH, is built on the opposite assumption. If Goudarzi et al. are right, the field has been measuring the wrong thing.
The Core Finding
The paper, “Math Reasoning in LLMs is Organized by Approach, Not Topic,” tests a specific hypothesis: that math-capable LLMs internally organize around reasoning process rather than problem category. To test it, the team developed what they call a “generation-replay protocol.” A model first generates a solution; that exact prompt-plus-generation trajectory is then replayed to extract activation-importance signatures across the reasoning tokens. Those signatures are clustered without supervision across Clemson’s eight target LLMs and five mathematical reasoning datasets, giving 40 model-source combinations in total.
The recovered clusters consistently beat random baselines of matched size. Two independent frontier-LLM judges found approach-level coherence in 77-82% of the real clusters, against 6-11% in within-source controls. The intervention test is the sharpest result: when the researchers changed the requested reasoning approach in approach-controlled prompts, cluster assignment shifted in seven of the eight model conditions. Paraphrasing the prompt, by contrast, largely preserved it. The internal structure tracks method, not topic.
What This Means for Benchmarks
Benchmarks such as MATH and GSM8K partition problems by domain: geometry, number theory, calculus. The Clemson finding suggests that even a training corpus deliberately balanced across those domains can still skew the underlying distribution of reasoning approaches. A model that looks strong at algebra may simply have extensive exposure to symbolic manipulation or multi-step equation solving, reasoning approaches that happen to dominate algebra problems. Topic-based accuracy scores would attribute that strength to the wrong cause.
The practical consequence is that targeted improvement becomes harder to identify. If a model underperforms on geometry, the diagnosis “needs more geometry data” may miss the actual gap, which might be a deficit in spatial case analysis or diagrammatic reasoning. An approach-centric benchmark design, one that explicitly tests backward induction, proof by contradiction or case analysis across domains, would surface those gaps more directly. The study does not propose a specific benchmark design, but the implication for how evaluation corpora should be structured is clear.
How LLMs Actually Tackle Math
The approaches the Clemson clusters appear to capture include several that the broader research community has studied independently. Chain-of-Thought prompting guides models to produce intermediate reasoning steps before a final answer, decomposing complex problems into manageable stages. Self-consistency decoding extends that by sampling multiple reasoning paths and taking the modal answer. Program synthesis takes a different route: the model generates executable code, offloads computation to an external environment and uses the result. Tool-augmented reasoning is a related variant, routing specific sub-problems to symbolic solvers while the model handles natural language interpretation.
One more recent development is the Iteratively Improved Program Construction method, introduced in February 2026, which refines programmatic reasoning chains through execution feedback and combines that with base CoT ability. The Clemson finding suggests that a model’s skill at tool-augmented reasoning may transfer across mathematical domains more reliably than its performance on any single topic, which is a different picture of what “math ability” actually means. This connects to broader work on how LLMs build internal world representations that generalise beyond their training distribution.
Training Implications
If approach is the organising axis, training data curation looks different. Diversity of reasoning strategy, not diversity of topic, becomes the lever. Rather than assembling corpora balanced across algebra, geometry and statistics, developers might prioritise datasets that demonstrate a range of reasoning methods applied across any domain: step-by-step decomposition, self-correction loops, proof structure, estimation under uncertainty.
There are precedents for this kind of approach-targeted optimisation. Microsoft Research, in work published in June 2025, identified improving architectural design and incorporating explicit mathematical reasoning techniques as primary strategies for strengthening smaller models’ reasoning capabilities. MIT researchers in December 2025 developed techniques for dynamically adjusting computational budgets by problem difficulty, which is itself an approach-level intervention. The Clemson paper gives those lines of work a unifying frame: they are all, in effect, optimising approaches rather than topics, even when that was not the stated objective.
Where Generalisation Still Breaks Down
The approach-organisation finding does not resolve the harder question of whether LLMs can genuinely generalise to novel mathematics. OpenAI’s reported solutions to a geometry conjecture in May 2026 and the Navier-Stokes problem in September 2026 drew scrutiny alongside the headlines. Critics, including researchers who commented publicly in September 2026, questioned whether those results represent novel reasoning or pattern-matching against the large volume of prior mathematical attempts embedded in training data. That debate is unresolved, and the Clemson paper does not settle it, the study characterises internal organisation, not whether that organisation produces genuine discovery.
What the paper does establish is that the field’s current measurement tools are not well-aligned with what models actually do internally. Whether approach-centric evaluation would also reveal the limits of generalisation more clearly is the next question, and one that harder benchmarks like FrontierMath are already starting to stress-test.
