Context Engineering for Agentic RAG: Caching and Cost Control

Overview

Single-shot RAG is well understood: retrieve once, generate once, cost is predictable. Agentic RAG behaves differently. The agent decides what to look up, reads the result, decides whether it needs more, and iterates — which means several retrievals per question and a context that grows with every step. The cost profile changes from a fixed amount per query to a variable amount that depends on how many steps the agent takes, and without deliberate control it grows faster than the value it delivers. Context engineering is the discipline that keeps it bounded.

Why the cost structure changes

In single-shot RAG the prompt is bounded: a system prompt, a fixed number of retrieved passages, the user question. Cost per query is therefore roughly constant and easy to plan against. In agentic RAG the trajectory is open-ended — the agent may retrieve twice, or six times, reformulating queries and reading results in between.

Two effects follow. Total tokens processed per question rise several-fold, since each step reprocesses the accumulated context plus the new material. And variance rises sharply: a simple question resolves in one step while a hard one takes eight, so capacity planning must consider the tail rather than the mean. Sizing on average step count consistently under-provisions.

KV cache is the binding constraint

Each active agent session holds attention cache for its whole accumulated context. Retrieved passages are long, and an agent that has read four documents carries all of them forward unless something removes them. Concurrency multiplies this: ten simultaneous sessions each holding 30,000 tokens of trajectory is a substantial memory commitment before the model weights are counted.

The observable failure is not a crash but degradation — the serving system evicts cache, recomputes, and latency rises for everyone. Because it presents as general slowness rather than an error, teams often respond by adding GPUs when the correct fix is bounding context. Measuring cache occupancy per session is the diagnostic that distinguishes the two.

Prefix caching

Prefix caching deserves the most attention because it is nearly free in quality terms. Agent prompts begin with the same system instructions, tool schemas and often the same organisational context on every request; caching that prefix’s attention state avoids recomputing thousands of tokens per step. Structuring prompts so stable content comes first and variable content last is what makes the cache hit.

Compaction without losing the thread

Compaction replaces verbatim history with a summary. Done crudely it destroys the agent’s ability to reason about what it has already learned; done well it is the difference between a system that scales and one that does not. Three practices help.

Summarise per step rather than in bulk: when a retrieval result has been used, replace the full passage with a short statement of what it established, retaining a citation so the source can be re-fetched if needed. Keep the original user question and the running plan verbatim, since those anchor the trajectory. And retain the most recent step in full, because the agent’s immediate next decision usually depends on detail rather than summary. The entry-tier version of this pattern is in running local agents on entry hardware.

Retrieval budgets

An agentic system needs explicit limits, and they should be product decisions rather than accidents of implementation. Useful budgets: maximum retrieval calls per question, maximum total context tokens, maximum reasoning steps, and a wall-clock timeout. When a budget is exhausted, the correct behaviour is to answer with what has been gathered and say so, not to fail silently or continue indefinitely.

Setting the numbers is empirical. Instrument a pilot, plot answer quality against step count, and find the point where additional steps stop improving answers — in most enterprise deployments that plateau arrives earlier than expected, often around three to five retrievals. Budgeting just above the plateau captures nearly all the quality at a fraction of the tail cost.

What this means for capacity planning

Three planning consequences. Size KV cache explicitly rather than treating it as a residual, using measured per-session occupancy at your typical trajectory length multiplied by target concurrency. Provision for the tail, since a small fraction of long trajectories consumes a disproportionate share of memory. And separate the retrieval tier from the generation tier so that embedding and search load does not contend with generation for the same accelerators.

The prefill-heavy character of agentic RAG — long contexts processed repeatedly — also makes it a strong candidate for the architecture in disaggregated inference, where prefill and decode are served by different pools. And the decision of when an agent should retrieve at all rather than rely on a long context is covered in long context vs RAG in 2026.

Frequently asked questions