Local RAG on a Mid-Tier Workstation: Index Size and Honest Limits

Overview

Running a private retrieval system entirely on a workstation is one of the more practical uses of mid-tier hardware, and one of the more commonly over-promised. The appeal for Indian teams handling confidential or regulated material is straightforward: nothing leaves the machine, so a whole category of governance questions never arises. The constraint is that three things compete for the same VRAM — the generation model, the embedding model and the vector index — and understanding that split is what separates a system that works from one that thrashes.

Where the memory goes

Start by separating the four consumers. The generation model dominates: a 32-billion-parameter model at 4-bit occupies roughly 18 GB. The embedding model is small — typically under 2 GB — and only needs to be resident during indexing and query embedding. KV cache holds the prompt, and RAG prompts are long because they carry retrieved passages, so this is larger than for plain chat. The vector index is the fourth, and it is the one people misplace.

Vector search does not require GPU residency at the scales a workstation handles. A quantized index over a few hundred thousand documents fits comfortably in system RAM and searches fast enough on CPU that the GPU is better used for generation. Putting the index in VRAM on a workstation is usually a mistake: it competes with the model for the scarcest resource in exchange for latency improvement the user cannot perceive.

What a workstation realistically holds

These figures assume moderate embedding dimensions and scalar quantization; the full arithmetic including graph overhead is in vector index sizing. The practical reading is that a well-specified mid-tier workstation with 128 GB of system RAM handles a corpus of a few hundred thousand documents — which covers most departmental knowledge bases, personal research libraries and single-project document sets.

The KV cache surprise

The most common memory failure in a local RAG deployment is not the index. It is that RAG prompts are long. A query that retrieves eight passages of 800 tokens each carries over 6,000 tokens of context before the user’s question, and if the system supports conversation history the prompt grows further with each turn.

Two consequences. Concurrency falls sharply: a workstation that would serve six chat sessions might serve two RAG sessions with the same model. And the temptation to retrieve more passages for better coverage has a direct memory cost. The disciplined response is to retrieve broadly but re-rank aggressively, passing only the best three to five passages to the generator — which improves quality and reduces memory simultaneously.

Re-ranking is the best value component

A cross-encoder re-ranker takes the top candidates from vector search and scores each against the query directly, which is far more accurate than embedding similarity. Re-rankers are small, often a few hundred million parameters, so they add little memory pressure, and they consistently produce the largest quality improvement per gigabyte of any component in the stack.

The pattern is: retrieve 30 to 50 candidates cheaply by vector similarity, re-rank them, pass the top few to the generator. On a workstation this is close to free and it addresses the main weakness of pure vector retrieval, which is that semantic similarity and actual relevance diverge more than people expect on technical corpora.

Honest limits

Three things a workstation RAG system does not do well. Concurrency: it serves one user comfortably, two at a stretch, and degrades badly beyond that because every session multiplies KV cache. Ingestion at scale: embedding a large document set is a batch job measured in hours, and re-embedding after a model change repeats it. Availability: it is a single machine, so any expectation of it being reliably available to others is misplaced.

It also cannot substitute for a properly engineered enterprise deployment where document permissions matter. A local system typically indexes everything the user can already see, which is correct for a personal system and unacceptable for a shared one where per-user access control must be enforced at retrieval time. That is a server-side concern, covered in right-sizing a GPU server for enterprise RAG.

When it is exactly the right tool

Three cases justify it clearly. Confidential single-user work — a lawyer, a researcher, an engineer working with material that must not leave the machine, which under DPDP is the simplest possible governance position. Development of a RAG system that will later run on a server, where building against a local stack gives fast iteration and the architecture transfers. And field or disconnected use, where no network is available.

For any of these, a mid-tier workstation with substantial system RAM, a 32 GB or larger GPU and fast NVMe is a genuinely capable platform. The tier’s broader capabilities are described in the 96 GB desk-side tier, and the routing question of when retrieval beats long context in long context vs RAG in 2026.

Frequently asked questions