KV Cache Offloading: The New Storage Tier in AI Inference Servers
Overview
The newest storage tier in AI infrastructure is the KV cache: as long-context and agentic workloads dominate 2026 inference, the intermediate attention state (the KV cache) for prior tokens has become too valuable to recompute and too large to keep in VRAM — so it spills to CPU RAM, NVMe and even shared network storage. Open-source layers such as LMCache (backing vLLM, SGLang and NVIDIA Dynamo) report 3–10× latency reductions on cache-hit-heavy workloads by persisting KV blocks across GPU, DRAM, disk and object storage, while NVIDIA’s NIXL transport and partner “KV-cache extender” appliances make cache movement a first-class datacenter function. For buyers, the consequence is concrete: inference servers now need a storage specification, not just a GPU count.
Key takeaways
- KV cache grows with context × concurrency: a 70B model at long context can consume more memory in cache than in weights — VRAM alone cannot hold it.
- Recomputing prefill is the alternative to storing cache; at 2026 context lengths, storing and reloading from NVMe is routinely faster and cheaper.
- The working hierarchy: hot cache in VRAM, warm in CPU DRAM, cold on local NVMe, shared/persistent in network storage — managed by layers like LMCache.
- Cache-aware serving lifts effective capacity: multi-turn chat, RAG with repeated system prompts, and agentic loops see the biggest wins (reported 3–10× TTFT gains on hits).
- Spec consequence: generous DRAM (1–2 TB), multiple high-endurance NVMe drives and fast fabric are now part of the inference bill of materials.
Why the cache outgrew the GPU
Every token in context holds KV state per layer; at FP16, a 70B-class model costs roughly 300–350 KB per token of cache, so a single 128K-token session approaches 40–45 GB — before concurrency. Fifty concurrent long-context sessions exceed any single GPU’s memory by an order of magnitude. The 2024-era answers (evict and recompute) became untenable as agentic workloads made contexts long, repetitive and revisited: an agent that returns to a session after a tool call re-pays the full prefill unless its cache survived somewhere. Handbook treatments like BentoML’s KV offloading guide frame the trade precisely: store-and-reload beats recompute whenever storage bandwidth exceeds the effective prefill rate — which modern NVMe does for most model sizes.
The tiering hierarchy in practice
Production stacks now run four cache tiers. VRAM holds active sessions’ hot blocks. CPU DRAM — the highest-leverage upgrade — extends cache 5–10× at microsecond access; engines’ native swap flags use it per-instance, while cache layers persist across restarts and share across replicas. Local NVMe holds cold sessions and long-tail prefixes; with GPUDirect Storage-class paths, reload bypasses the CPU entirely. Shared storage (Redis/Mooncake/S3-class backends over NIXL-style transports) makes caches portable across a serving fleet — the architectural unlock for disaggregated prefill/decode designs, where the emerging extender-appliance ecosystem competes on exactly this movement. The scheduling logic — which blocks live where — is the serving engine’s job; the buyer’s job is provisioning the tiers.
What it changes on the spec sheet
Three line items move. DRAM: inference nodes historically shipped RAM-light; cache offload inverts that — 1–2 TB of system memory on an 8-GPU inference node is now defensible, often the cheapest “VRAM extension” available. NVMe: capacity is easy, endurance and write bandwidth are not — KV traffic is write-heavy and continuous, so specify datacenter-class drives (high DWPD) and multiple drives for parallelism rather than one large consumer SSD. Fabric: fleets sharing cache need the same class of network attention as training clusters — a theme continuous with our checkpoint and RAG index storage guide, which covers the adjacent tiers. Buyers using the inference sizing checklist should add cache tiering as an explicit line.
Which workloads pay back the engineering
Cache hit rate decides everything. Multi-turn assistants (every turn re-uses the conversation prefix), RAG at scale (system prompts and shared document prefixes repeat across users — see the RAG reference architecture), agentic loops (sessions suspend during tool calls and resume), and code assistants (repository context repeats per keystroke burst) all sit at high hit rates where 3–10× TTFT improvements are realistic. One-shot, unique-prompt workloads — batch classification, single-question answering — gain little; skip the complexity there. For Indian deployments the economics are amplified: cache offload raises served users per GPU, which is the binding constraint when accelerators are the scarce, imported line item and DRAM/NVMe are comparatively cheap.
Cache-tier sizing frame
| Tier | Capacity guide | Access | Holds |
|---|---|---|---|
| VRAM (per GPU) | What remains after weights | ns–µs | Active session hot blocks |
| CPU DRAM | 1–2 TB per 8-GPU node | µs | Warm sessions, shared prefixes |
| Local NVMe | 10–30 TB, high-DWPD, multiple drives | tens of µs–ms | Cold sessions, long-tail cache |
| Shared store | Fleet-wide, workload-dependent | ms over fabric | Cross-replica and persistent cache |
Frequently asked questions
How big does a KV cache actually get?
Roughly 300–350 KB per token for a 70B-class model at FP16 — about 40 GB for one 128K-token session, scaling linearly with concurrency. Quantised cache (FP8/INT4 KV) halves-to-quarters this at modest quality cost.
Is reloading cache from NVMe really faster than recomputing?
For long contexts, usually yes: prefill is compute-expensive and serialised, while NVMe reload streams at GB/s. The crossover favours storage more strongly as context grows — which is the 2026 trend line.
Do we need special appliances for this?
No — open layers (LMCache-class) over generous DRAM and datacenter NVMe cover most deployments. Extender appliances and NIXL-based shared tiers earn consideration at fleet scale with disaggregated serving.
What NVMe endurance should we specify?
KV traffic is continuous and write-heavy: datacenter drives at 1–3 DWPD, spread across multiple drives for bandwidth. Consumer SSDs wear out and throttle — a false economy under this workload.
Does cache offload help short-prompt workloads?
Minimally — hit rates are what pay. Multi-turn, RAG, agentic and code workloads benefit most; unique one-shot prompts see little and don’t justify the added moving parts.
Ready to deploy?
Talk to an RDP architect about power, cooling and lead time.