KV-Cache Reuse and TTFT: Why Inference Latency Gets Unstable at Scale

Overview

Inference benchmarks usually quote a single latency number measured on an idle machine. Production behaves differently: as concurrency rises, time-to-first-token (TTFT) becomes unstable, and eventually requests start failing. The mechanism is the KV cache — the stored attention state that lets a model avoid recomputing the prompt on every token. When GPU memory runs short, that cache is evicted and must be recomputed, and latency falls off a cliff. This article explains the failure mode and what published aiDAPTIV+ figures show when the cache is retained instead.

What the KV cache actually is

When a transformer processes a prompt, it computes key and value tensors for every token in every attention layer. Retaining those in the KV cache means each new token attends to stored state instead of reprocessing the whole prompt. The cache is therefore what makes generation fast — and it is large. On a reference 8x RTX PRO 6000 Blackwell node (768 GB), weights load once and are sharded across the GPUs; the remaining memory holds the KV cache. That residual capacity, not the weights, determines how many concurrent requests the node can genuinely serve.

The eviction cliff

Cache size grows with both context length and concurrency. Add enough simultaneous sessions and the memory left after weights is exhausted. The server then evicts cache entries to make room — and any request whose entries were evicted must recompute its prompt from scratch on the next token. This is why warm TTFT is published as a range rather than a number: 190-340 ms, depending on whether that particular request’s cache survived. Push further and the behaviour degrades from variable to broken, with the source describing hits to OOM and failed requests.

The published comparison

Why the cold prompt is slower — and why that is fine

Note the one row where caching loses: the first cold prompt is published at 826 ms versus 769 ms. Filling the cache costs a little extra on the very first pass. That is an honest trade and the right one, because the cold path happens once per prefix while the warm path happens on every subsequent request. Paying 57 ms once to make every following request stable at 78 ms instead of a variable 190-340 ms is straightforwardly worth it in any sustained workload.

Designing for stable latency

Three practical implications follow. First, benchmark under concurrency, not at idle — a single-user TTFT figure tells you almost nothing about production. Second, size for the KV cache, not just the weights: a model that fits in VRAM but leaves no headroom will serve very few concurrent users. Third, quote p95 or p99 rather than average, because a 250 ms average built from a 190-340 ms spread is a different service from a stable 78 ms. Where prompts share long common prefixes — a system prompt, a retrieved document set, a chat history — prefix reuse is especially valuable, which is why it pairs well with the retrieval patterns covered in our aiDAPTIV+ architecture explainer.

What this means for Indian deployments

Latency stability is a customer-experience and compliance issue as much as an engineering one. A citizen-facing service, a bank’s assistant or a hospital’s clinical tool cannot present a response time that swings by a factor of two depending on how busy the system is, and failed requests under load are worse still. Serving on-premises adds the data-residency benefit: prompts and retrieved context frequently contain personal data, and under the DPDP Act — penalties up to ₹250 crore per violation — keeping that content inside your own infrastructure avoids the transfer question entirely. Validate the numbers above against your own model and context length in a scoped proof-of-concept.

Frequently asked questions