New LAPS System Cuts LLM Prefill Latency Over 30% with Load-Aware Deflection

New LAPS System Cuts LLM Prefill Latency Over 30% with Load-Aware Deflection
Key Takeaways

  • LAPS, presented at MLSys 2026, cuts prefill latency by more than 30% and reduces service-level objective (SLO) violations by 28% compared to vanilla SGLang under prefill-decode disaggregation.
  • LAPS separates long and short prefill requests into distinct processing pools using a dual-queue design, eliminating the compute-memory interference that unified scheduling leaves unresolved.
  • In high-concurrency tests with Qwen2.5-32B, LAPS improved request throughput by 35%, meaning the same GPU cluster can serve materially more users without added hardware cost.

Most LLM serving research focuses on splitting prefill and decode across separate GPUs. LAPS, presented at MLSys 2026, goes a level deeper: it splits the prefill stage itself, routing long and short prompts to separate processing pools. The result is a more than 30% drop in prefill latency and 35% more throughput on models like Qwen2.5-32B, without touching the underlying model.

Addressing LLM Serving Bottlenecks

LLM inference has two distinct phases with almost nothing in common. Prefill processes the entire input prompt in parallel to compute the Key-Value (KV) cache; it is compute-bound and scales with prompt length. Decode then generates output tokens one at a time using that cache; it is memory-bandwidth-bound and relatively stable per token. When both phases share the same GPU, a long prefill can stall every decode operation behind it, a classic head-of-line blocking problem that gets worse as prompts grow longer and multi-turn conversations pile up re-prefills.

Traditional serving systems colocate prefill and decode on the same GPUs, batching across requests to maximise throughput. That works at low concurrency, but under load the interference becomes costly. Time to first token (TTFT), which prefill directly controls, and time per output token (TPOT), which decode controls, pull against each other. Over-provisioning GPU capacity is the standard fix, and it is an expensive one.

LAPS Improves Prefill Latency with Disaggregation

“LAPS: A Length-Aware-Prefill LLM Serving System,” detailed in an arXiv paper revised in January 2026, takes a different approach. Rather than treating all prefill requests as a single queue, it disaggregates them by prompt length, routing long prefills and short prefills into separate pools. The paper’s central observation is straightforward: a 4,000-token multi-turn re-prefill and a 200-token single-turn query have fundamentally different compute profiles, and scheduling them together creates interference that no amount of batching can fully resolve.

Separating them does. Against SGLang running vanilla prefill-decode disaggregation, LAPS cuts prefill latency by more than 30% and reduces SLO violations by 28% in multi-instance deployments. For short-prefill workloads specifically, LAPS adds a smart batching layer: a batch waiting window combined with CUDA Graph-based clustering groups requests by compute similarity, reducing batching delay and keeping average latency low even when request arrival is irregular.

The Architecture of Load-Aware Prefill Deflection

The dual-queue design is what makes both modes work. A single prefill instance can handle temporal disaggregation, processing long and short prefills in sequence without blocking. Across multiple instances, the LAPS scheduler handles spatial disaggregation, shifting each instance’s workload type in real time based on current load and incoming prompt characteristics. The paper calls this “prefill deflection”: routing a long prefill away from an instance already saturated with short requests, and vice versa.

The comparison point here is DistServewhich disaggregates prefill and decode to separate GPUs entirely and has shown it can serve up to 7.4 times more requests or meet SLOs up to 12.6 times tighter than colocated serving, according to its published results. LAPS accepts that disaggregation baseline and adds a second layer of granularity within the prefill side. Where DistServe separates phases, LAPS separates lengths. The two ideas are composable.

Impact on Throughput and Cost Efficiency

In high-concurrency tests serving Qwen2.5-32B across mixed request types, LAPS improved throughput by 35%, according to the paper’s benchmarks. For an operator running a fixed GPU cluster, that means roughly a third more capacity without additional hardware. The cost implication is direct: each inference becomes cheaper as the denominator grows.

Meta has explored prefill-decode disaggregation at production scale for similar reasons, separating compute-heavy prefill from memory-heavy decode so each can be optimised independently. LAPS builds on that direction by introducing finer control within the prefill phase, which matters most in multi-turn applications where re-prefills with short prompts are the dominant traffic pattern. Getting those short re-prefills out of the way quickly, rather than queuing them behind long context windows, is where much of the practical throughput gain comes from. How widely these results generalise across different model families and serving configurations is harder to verify from the paper alone, but the mechanism is sound.

Future Directions in LLM Serving

LAPS is part of a broader move toward modular, disaggregated serving architectures. NVIDIA‘s Dynamo 1.0 already applies disaggregated design at scale, adding a routing layer on top of frameworks like vLLM to handle multiple distinct LLMs on shared GPU clusters. A separate research system called TaiChi takes a complementary angle: rather than fixing which mode a system runs in, it switches between prefill-decode aggregation and disaggregation dynamically, and introduces “latency shifting” to reallocate GPU resources toward requests approaching an SLO breach.

The direction across all of these systems is the same: away from static resource allocation and toward schedulers that respond to what is actually happening in the queue. LAPS demonstrates that prompt length alone is a signal worth acting on. Whether that insight gets absorbed into mainstream serving frameworks or remains a research-track optimisation will depend on whether the operational complexity of running separate prefill pools is worth the throughput gain for a given workload. For teams running high-concurrency multi-turn applications on large models, the paper suggests it probably is. For more coverage of AI research and breakthroughs, visit our AI Research section.

Taylor Voss
Taylor Voss

Taylor Voss is a research correspondent covering AI breakthroughs, model releases, and the science shaping the future of artificial intelligence. Taylor translates complex research into clear, compelling stories for curious readers who want more than the headlines.

📰 Journalists welcome — cite Auton AI News with attribution. Press & Media → | press@autonainews.com