- A study of 120 prompt-caching pull requests from 10 open-source agent systems found that most mis-integrations fail silently, without runtime errors, according to a paper posted to arXiv on October 26, 2026 and accepted at ICSE 2027.
- The researchers’ static analysis tool, CacheLint, found 32 previously unknown mis-integrations across 571 agent projects at 0.86 precision, leading to 21 developer confirmations and 18 upstream fixes, including one in Anthropic’s official SDK.
- Separately, ProjectDiscovery said it raised its cache hit rate from 7% to 84% by moving dynamic working memory out of its cached system prompt, cutting overall LLM costs by 59%.
Prompt caching lets providers reuse processed prompt content across calls to cut inference cost and latency, but integrating it correctly is difficult, according to a paper posted to arXiv on October 26, 2026 and accepted at ICSE 2027. The paper, “Silent and Costly: Characterizing and Detecting Prompt-Caching Mis-Integration in LLM Agents,” by Ying Zhang and six co-authors, finds that most mis-integrations in agent applications cause no observable runtime failure.
What the Study Found
The researchers say correct integration is hard because providers use different caching APIs and their caching behaviour keeps changing. They describe their work as the first empirical study of prompt-caching mis-integration in downstream applications, examining 120 prompt-caching pull requests from 10 popular open-source agent systems. They found that mis-integrations arise in two forms. Of the 120 defects, 97 (81%) were on the request side, where the application corrupts the outgoing request, and 23 (19%) were on the read side, where requests stay intact but the application misparses cache-usage numbers, so caching works but the reported token counts are wrong. The authors grouped the defects into eight root causes, and the fixes were largely predictable: assigning each root cause its most common fix correctly predicted 95 of the 120 fixes, or 79%.
How CacheLint Works
The team built CacheLint, which treats a mis-integration as a caching call together with the conditions required for it to work correctly. It locates caching calls with CodeQL and checks each condition either against a dated record of provider capabilities or with an LLM guard check. On the 120 labelled mis-integrations from the ten studied systems, CacheLint scored 0.71 precision and 0.78 recall.
CacheLint’s preconditions are derived from the fixing patterns in that empirical study. Precision was lower on the labelled set (0.71) than on the real-world projects (0.86), and a recall of 0.78 means CacheLint missed roughly a fifth of the labelled defects.
Results on Real Projects
Run on 571 real-world agent projects, CacheLint identified 32 previously unknown mis-integrations at 0.86 precision. The findings produced 21 developer confirmations and 18 upstream fixes, including one in Anthropic’s official SDK, according to the paper.
What the Results Do Not Show
The paper is a first version posted to arXiv. Its real-world evaluation reports the defects CacheLint found, the developer confirmations and the upstream fixes, and the abstract does not give a dollar figure for the waste those defects caused. The empirical study covers open-source agent systems only.
A Separate Real-World Case
ProjectDiscovery, which builds an autonomous security testing platform called Neo, described a related problem in its own engineering blog earlier this year, separate from the paper. The company said Neo routinely runs 20 to 40 or more LLM steps per task, and that dynamic working memory was leaking into its cacheable system prompt. Moving it to the end of the prompt lifted the cache hit rate to about 74%, and smaller optimisations took it to 84%. It said overall LLM costs fell 59% compared with what the same token volume would have cost at full input rates.
