- Aditi Patodiya’s paper, submitted to IEEE Access in September 2026, finds that 4-bit quantization amplifies cache-induced output divergence from 36.2% to 75.0% in agentic tool-use workloads, more than doubling the rate of non-reproducible results.
- Prefix caching in serving stacks like vLLM and llama.cpp is the primary driver: disabling it restores bit-identical outputs, but the cache state is neither exposed nor reset by default, making divergence invisible to operators.
Three quarters of the time, a 4-bit quantized LLM agent given an identical prompt will take a different path than it did on the previous run. That figure comes from a paper by Aditi Patodiya, submitted to IEEE Access on September 4, 2026, and it puts a precise number on a problem that has been quietly undermining production AI deployments: prefix caching, the standard efficiency optimisation baked into serving stacks like vLLM and llama.cpp, interacts with quantization to make agent outputs effectively non-reproducible.
4-Bit Precision, Double the Instability
The core finding is stark. At 16-bit precision, cache-induced divergence affected 36.2% of agentic episodes in Patodiya’s 80-episode multi-turn benchmark. Drop to 4-bit quantization and that figure climbs to 75.0%, meaning the majority of runs produce a different outcome than the previous identical run. The memory and throughput case for 4-bit quantization remains real, but Patodiya’s measurements reframe the trade-off: teams optimising for inference economics need to account for what that precision loss does to output stability, not just to benchmark accuracy.
Prefix Caching Is the Mechanism
Prefix caching works by reusing the key-value tensors computed from shared prompt prefixes across requests, avoiding redundant computation. The efficiency gain is genuine. The problem, as Patodiya’s controlled experiments show, is that reusing cached tensors changes the order in which floating-point accumulations are performed, and floating-point arithmetic is non-associative. Because (a + b) + c and a + (b + c) produce different results at finite precision, recomputing versus retrieving from cache yields numerically distinct attention outputs. Quantization amplifies these small numerical gaps until they cross token-selection thresholds, producing a different next token and, from there, a different trajectory entirely.
When Patodiya disabled caching across model families and serving implementations, repeated executions were bit-identical. That result rules out other common sources of non-determinism and pins the divergence directly on the cache interaction. For ML engineers already familiar with how subtle LLM failure modes resist detection this is a particularly difficult one: the serving stack behaves correctly at every step, yet the system as a whole is unreliable.
Cache State: The Invisible Variable
Cache-enabled serving is, in theory, deterministic given a fixed cache state. The operational problem is that the cache state is neither reported by serving engines nor reset between requests by default. An application sending identical prompts to the same endpoint can receive different outputs with no change in input, no error, and no signal that anything is wrong. The divergence is silent.
For enterprises in regulated industries, that invisibility is the real risk. Debugging an agent that behaves inconsistently requires knowing what the server’s cache state was at the time of each run, information that most default deployments do not expose. Auditing outputs for compliance, or guaranteeing consistent behaviour across repeated calls, becomes structurally difficult without explicit cache management tooling. This connects to a broader pattern in multi-agent system reliability: individual components can pass every unit test and still produce unreliable system-level behaviour.
Agents Amplify the Problem
Single-turn tasks absorb divergence relatively gracefully. Agentic workloads do not. A single divergent token early in a multi-turn trajectory can redirect a tool call, which changes the data the agent receives, which changes every subsequent decision. Patodiya’s 80-episode benchmark captures this cascade: the divergence rate at 75.0% for 4-bit models reflects not just token-level differences but complete trajectory changes. Teams building agents for automated research, code generation or customer operations face a harder reliability problem than their benchmark numbers suggest.
Where Aggregate Metrics Lie
Perhaps the most useful finding for teams evaluating production readiness is what aggregate accuracy hides. Patodiya documents cases where cache-induced divergence shifts individual outputs in both directions, some previously correct answers become wrong, some wrong answers become correct, without moving the overall accuracy score. The paper calls this “instability without directional bias.” A model that scores identically across two evaluation runs is not necessarily producing consistent outputs; it may be producing different wrong and right answers that happen to cancel out.
This has direct implications for how teams evaluate LLM agents beyond aggregate benchmarks. An aggregate accuracy score can look stable while per-request behaviour is highly variable, which is exactly the condition that creates compliance exposure and unpredictable user experiences. Patodiya’s work makes a clear case for per-request consistency metrics alongside the averaged figures that currently dominate evaluation practice.
What Teams Can Do Now
Patodiya’s paper does not propose a single fix, but the experimental design points toward the levers available. Disabling prefix caching restores bit-identical outputs at the cost of the efficiency gains caching provides. Explicit cache state management, exposing and resetting cache between requests where reproducibility matters, addresses the invisibility problem without abandoning caching entirely. For teams running 4-bit quantized models in agentic pipelines, the paper’s practical upshot is that reproducibility needs to be tested directly, not assumed from accuracy scores, and that the serving configuration is as much a part of the system’s behaviour as the model weights themselves.
