Disaggregated Inference: Splitting Prefill and Decode in 2026

Overview

The biggest serving-architecture shift since continuous batching is disaggregation: running prefill (prompt processing) and decode (token generation) on separate GPU pools. The physics is simple — prefill is compute-bound, decode is memory-bandwidth-bound, and a GPU doing both does neither optimally. In 2026 prefill/decode disaggregation is supported across every major serving stack (vLLM, SGLang, TensorRT-LLM, and NVIDIA’s Dynamo orchestration layer, GA since March 2026), with reported gains from a reliable 2–3× throughput on multi-GPU vLLM deployments to vendor-benchmark peaks far higher on rack-scale reasoning workloads. This article explains when disaggregation pays, what it demands from hardware, and how smaller estates should think about it.

The physics: two workloads wearing one name

Prefill processes the entire prompt in parallel — large matrix multiplies, tensor cores saturated, memory bandwidth loafing. Decode generates one token at a time per sequence — every step re-reads the model weights and KV cache, bandwidth saturated, compute loafing. Explainers like Towards Data Science’s treatment capture the consequence: on a shared GPU, an arriving long prompt stalls in-flight decodes (time-between-tokens spikes), and decode-heavy batches waste the compute that prefill needs. Chunked prefill — slicing prompts into interleaved pieces — softens the interference and is the right first tool on small deployments; disaggregation eliminates it.

How the disaggregated design works

Three components. Worker pools: prefill workers ingest prompts and produce KV caches; decode workers hold sessions and stream tokens. KV transfer: the produced cache ships from prefill to decode workers — over NVLink within rack-scale domains, RDMA fabrics between nodes, or via shared cache layers (the LMCache/NIXL machinery described in our KV-cache article). A router: assigns requests cache-aware — sending a follow-up turn to the worker already holding its history — which is where orchestration layers like Dynamo live above vLLM-class engines. The design also lets operators scale pools independently: agentic and RAG traffic with huge prompts wants more prefill; chat with long generations wants more decode.

When it pays — and when it’s over-engineering

Honest thresholds from production practice: disaggregation earns its complexity when a model is served across ~8+ GPUs with high concurrency, when prompts are long relative to generations (RAG and agentic traffic — exactly the profile in adaptive RAG), and when p99 time-between-tokens is an SLO. Below that — the single-node estates most Indian enterprises run — continuous batching plus chunked prefill on a well-tuned vLLM captures most of the win, and the sizing checklist arithmetic stands unchanged. Vendor headline numbers (up to 30× served requests for a 671B reasoning model on GB200 NVL72 with FP4 disaggregated serving) are real benchmarks of the largest configurations — treat them as what rack-scale NVLink domains enable, not what an 8-GPU node should expect; 2–3× is the defensible planning figure at node scale.

Hardware implications, present and next

Present: disaggregation raises the value of fast interconnect (it is another argument for NVLink-domain systems on big serving estates — see rack-scale vs scale-out) and of DRAM/NVMe cache tiers on ordinary nodes. Next: phase-specialised procurement — prefill pools built from compute-dense parts and decode pools from bandwidth-rich (HBM) parts, including mixed-generation fleets where last year’s GPUs decode economically behind this year’s prefill tier. That pattern is how large serving operators are already extending fleet life, and it changes the depreciation story for owned estates: an H100-class node displaced from training duty becomes a first-class decode pool, not scrap.

Deployment tiers

Frequently asked questions