Running Local Agents on Entry Hardware: Limits and Workarounds

Overview

Running an agent locally is a different problem from running a chatbot locally. A chat turn is one call with a bounded context; an agent plans, calls tools, reads results and iterates, so context grows with every step and the number of model invocations multiplies. On an entry workstation with 24 to 32 GB of VRAM, the constraint that bites is not model size — it is KV cache growth across a long agent trajectory. This article covers what actually works at that memory budget, which patterns make it viable, and where the honest limit sits.

Why context growth is the binding constraint

Model weights are a fixed cost: a 14-billion-parameter model quantized to 4-bit occupies roughly 8 GB, leaving comfortable headroom on a 24 GB card. The variable cost is the KV cache, which scales with context length and grows monotonically through an agent run as each tool call and its output are appended.

A concrete illustration: a research agent that reads five documents, each contributing several thousand tokens of retrieved text, plus its own reasoning at each step, can reach tens of thousands of tokens of context before it produces a final answer. The KV cache for that context can rival or exceed the weights in memory. Teams sizing on “the model fits” are sizing the wrong thing, and the failure mode — an out-of-memory error twelve steps into a run — appears only in realistic testing.

What model sizes are practical

The dual-card row points at a pattern worth knowing: rather than one large model, run a small fast model for routing and routine steps alongside a larger one for difficult reasoning. That uses memory more efficiently than a single model sized for the hardest step, and it is the local analogue of the tiering used in production serving. The dual-card planning detail is in dual-GPU AI workstation planning.

Tool-calling reliability is size-dependent

The capability that makes an agent work is structured tool invocation: producing correctly formatted arguments for the right tool at the right moment. This degrades sharply as models get smaller. Below roughly 7 billion parameters, malformed JSON arguments, invented tool names and failure to recognise when a tool is needed become frequent enough to break workflows, regardless of how good the model looks on chat benchmarks.

Two mitigations help materially. Constrained decoding — forcing output to match a schema — eliminates malformed arguments entirely and lets smaller models be useful for routine calls. And keeping the tool surface small: an agent offered four well-described tools chooses correctly far more often than one offered twenty. Both are cheap changes that move the practical floor down by several billion parameters.

Patterns that make local agents viable

Four techniques do most of the work. Context compaction: after each tool call, summarise the result rather than appending it verbatim, and drop older steps once their conclusions are captured. This alone often halves peak KV cache. Difficulty routing: classify each step and send routine ones to a small model, reserving the large model for planning and synthesis.

Prompt caching: agent trajectories share a long common prefix — system prompt, tool definitions, early steps — so caching the prefix KV avoids recomputing it on every call, which is a substantial saving in both latency and compute. And bounded loops: cap the number of steps and tool calls explicitly, both to control memory and because an unbounded local agent that goes wrong will consume the machine. The production versions of these patterns are covered in GPU sizing for agentic AI workloads.

Where local agents genuinely make sense

Three cases justify the effort. Development and debugging: building an agent against a local model gives fast iteration without per-token cost, and the architecture transfers to a larger model later. Private data work: an agent that reads local files, code or documents on a machine that sends nothing outward is a much simpler governance position for regulated Indian workloads than any hosted alternative.

And single-user coding assistance, where a 14-32B model with good tool calling handles a large share of routine work at acceptable latency — the pattern developed in self-hosted coding assistants on entry workstations. What does not work is serving a team: concurrency multiplies KV cache, and a workstation running three simultaneous agent sessions will exhaust memory long before it exhausts compute.

Knowing when to stop

Three signals indicate the workstation is the wrong tool. Out-of-memory failures that persist after compaction and quantization, which means the trajectory genuinely needs more cache than the card holds. Latency that makes the agent unpleasant — a six-step agent at eight seconds per step is a 48-second wait, and users abandon it. And any requirement for more than one concurrent user.

At that point the choice is between a larger desk-side machine and server hardware. The honest guidance is that agent workloads scale poorly on the desk because concurrency is their natural growth direction, so the graduation tends to come sooner than for single-model chat use. The transition criteria are set out in when a team outgrows AI workstations.

Frequently asked questions