In-Process Retrieval Cuts LLM Latency by 83x in New ‘Memory in the Loop’ Design

Most AI agents feel slow during complex tasks because their memory is architecturally in the wrong place. A paper posted to arXiv by Yusuf Khan and Carlo Lipizzi argues that moving retrieval inside the reasoning loop, rather than bolting it on as an external call, largely bypasses the 83x latency penalty that plagues conventional RAG systems. That is not a narrow engineering fix; it changes what kinds of tasks agents can realistically attempt.

The Latency Bottleneck: Why Memory Moved Outside the Loop

Language agents run on a repeating observe-reason-act cycle. “Observe” has always included a trip to an external memory store, and that trip costs time. Networked vector databases, the backbone of most Retrieval Augmented Generation (RAG) systems, respond in tens to hundreds of milliseconds per query. That sounds negligible until you count how many queries a multi-step reasoning chain requires. The delays compound, and end-to-end latency can swell by up to 83 times when retrieval is computationally intensive, according to the paper.

Developers have managed this two ways: hiding latency through retrieval scheduling, or rationing memory access to once per conversational turn. Both are compromises. The once-per-turn approach is more common, and its cost is real: the agent finishes an entire reasoning step, pauses to consult its knowledge base, then continues. Memory and thought are sequential, not interleaved. For tasks requiring continuously updated context, that disjointed rhythm is a ceiling, not just an inconvenience.

In-Process Retrieval: A New Architecture for Agent Memory

Khan and Lipizzi’s proposal is to move the memory store inside the agent’s process, local to the LLM’s computation, so that reading from and writing to it costs roughly what consulting the existing context window costs. No network round-trip. No API call. They call this “in-process retrieval,” and it shifts retrieval frequency from per-turn to per-step.

Per-step matters more than it sounds. An agent can now retrieve a relevant fact mid-reasoning, update its internal state based on that fact, and immediately retrieve again based on the updated state, all within a single reasoning cycle. Traditional RAG cannot do this cheaply: each additional retrieval in a conventional setup adds another network hop, while here the cost is essentially flat. The authors describe this as an ephemeral, in-process semantic store, fast enough to participate in active thought and not persisted across sessions by design.

The reproduction artifact accompanying the paper is worth noting separately. Khan and Lipizzi release a Docker build, experiment scripts and per-run JSON records alongside the arXiv posting. The architecture is testable now, without waiting for third-party replication. That level of methodological transparency is rarer than it should be in agent research.

The Parity Principle

Alongside the architecture, Khan and Lipizzi introduce a conceptual framing they call the “Parity Principle.” Their claim: memory qualifies as a genuine cognitive resource only when it is accessible with the same fluidity as information already in the context window. Anything slower is an external reference tool, not working memory.

Most current agent designs treat memory as a reference library, something consulted at defined checkpoints rather than something actively shaping each sub-step of reasoning. The Parity Principle argues that is the wrong model. If retrieval is expensive, you ration it. If rationing it hurts complex tasks, the solution is to make retrieval cheap enough that rationing becomes unnecessary. In-process storage is how they get there.

The analogy to human cognition is inexact but instructive. Working memory in human reasoning is not a database call; it is a continuously active resource that shapes thought as it unfolds. The “Memory in the Loop” architecture is an engineering approximation of that idea, not a neuroscience claim. The practical value is this: agents that can interleave retrieval and reasoning per-step can, in principle, handle longer-horizon tasks without fragmenting their chain of thought.

How This Compares to External RAG and Fine-Tuning

External RAG is the dominant memory strategy for production agents today. It scales well, it is interpretable, and it leaves the model itself unchanged. The problem is the network hop. Caching and parallelisation reduce the cost but cannot eliminate it, and the per-turn retrieval rhythm remains a structural constraint.

Fine-tuning is the other common approach: bake knowledge directly into the model’s weights, eliminating retrieval overhead entirely. The cost is flexibility. Updating fine-tuned knowledge means retraining, which is slow and expensive. Encoded knowledge is also opaque: difficult to inspect, harder to correct. For agents operating in domains where information changes frequently, fine-tuning is poorly suited.

A related line of research pursues a middle path: episodic and semantic memory structures that let agents learn from feedback without touching the model weights, aiming for continuous learning at inference time with reduced token overhead. Another approach takes a different angle altogether, modelling long-term memory management on biological mechanisms including something analogous to sleep-phase consolidation and interference-based forgetting. Khan and Lipizzi’s work does not compete with that line so much as operate at a different timescale.

What This Means for Agent Design

Debugging a complex codebase is a useful concrete case. It involves continuous cross-referencing: error logs against documentation, function signatures against call stacks, hypotheses against test results. With per-turn RAG, each cross-reference is a pause. With in-process retrieval, the agent moves through those references continuously, writing tentative conclusions to its working memory and revising them as new facts emerge, without leaving the reasoning loop. The same logic applies to multi-turn planning, real-time monitoring and tasks requiring iterative self-correction.

For AI-assisted code work specifically, where latency directly affects whether a tool feels usable, the architectural change matters beyond benchmark numbers.

There is also an auditability angle. Because the agent can write intermediate states to its in-process memory, there is a natural record of its reasoning path: the hypotheses it held, the facts it retrieved, the corrections it made. That kind of interpretable internal record is difficult to produce from a model operating on a static context window.

Open Questions

Khan and Lipizzi are direct about what the paper does not solve. Making retrieval cheap per-step is a necessary condition for in-loop memory; it is not sufficient. Two harder problems remain: what an agent should retain, and how it should organise what it retains.

An agent with per-step write access to its working memory can accumulate noise as fast as it accumulates useful information. Without principled selection, the store degrades. The signal-to-noise problem that plagues external RAG does not disappear when retrieval moves in-process; it shifts from the query-response interface to the write-selection interface. Solving it will require either learned heuristics or explicit meta-cognitive components that assess what is worth retaining at each step.

Scalability is the other constraint. In-process memory must remain compact enough not to introduce its own computational overhead. For short reasoning chains this is straightforward; for extended tasks, the ephemeral store could grow large enough to create a different kind of bottleneck. Compression, selective forgetting and efficient data structures are the obvious engineering levers, but none are solved problems in the context of live agent reasoning.

The longer-term integration challenge is bridging in-process working memory with persistent external stores. Ephemeral by design, in-process memory does not survive session boundaries. Anything the agent needs to carry forward, a user’s preferences, accumulated project knowledge, learned corrections, still requires externalisation. The research question is how to make that handoff fast enough that it does not reintroduce the latency the in-process architecture was designed to avoid. This is where recent agent infrastructure developments from major labs become relevant, since the tooling for managing that boundary barely exists yet.

What to Watch Next

The near-term signals worth tracking are specific. First, whether open-source agent frameworks, LangChain and its successors being the obvious candidates, incorporate native in-process memory components. The architecture is not proprietary; adoption depends on whether the engineering overhead is low enough for framework maintainers to absorb.

Second, benchmarks. The paper makes a latency argument; what is still needed is evidence that per-step retrieval translates into measurably better task completion on long-horizon problems, not just faster individual queries. The research community’s answer to the memory retention and organisation questions will largely determine how quickly that evidence arrives.

Third, and less obvious, is the memory management problem itself. The teams that crack principled, automated selection of what to retain in working memory will have solved something that limits both in-process and external architectures. That is the harder contribution. The Khan and Lipizzi paper frames the problem clearly, which is often the more valuable step. For more coverage of AI research and breakthroughs, visit our AI Research section.