Vector Index Sizing: From 10 Million to 1 Billion Embeddings

Overview

Vector index sizing goes wrong in a predictable way: a team prototypes on a few million embeddings in memory, the corpus grows tenfold, and the index no longer fits on any single machine at a sane price. The arithmetic is unforgiving — a billion 1536-dimension float32 vectors is roughly 6 TB of raw vectors before graph overhead, and an in-memory HNSW index on top of that needs terabytes of RAM. This article gives the sizing method, the compression options that change the answer, and the architecture that applies at each scale.

The raw arithmetic

Begin with uncompressed vectors: count multiplied by dimensions multiplied by 4 bytes for float32. At 1536 dimensions that is about 6.1 KB per vector, so 10 million vectors is roughly 61 GB and 1 billion is roughly 6.1 TB before any index structure. For HNSW, a common working estimate is to take that raw figure and roughly double it to account for graph links and allocator overhead. One published rule of thumb puts 5 million rows of 1536-dimension vectors at around 60 GB with graph overhead included, which is consistent with that doubling.

Two adjustments matter. Dimension choice is a first-order lever — moving from 1536 to 768 dimensions halves everything, and many embedding models now support Matryoshka-style truncation with modest quality loss. And metadata, payloads and the original text usually live alongside the index; budget them separately rather than discovering them at deployment.

What quantization buys and costs

Quantization is where the sizing problem becomes tractable. Scalar quantization stores each dimension in 8 bits instead of 32, giving roughly a 4x reduction with minimal recall impact for most embedding models — this is close to a free win and should usually be the default. Binary quantization reduces to one bit per dimension for roughly 32x, with moderate recall loss that is often recoverable by re-ranking a larger candidate set against full-precision vectors.

Product quantization splits vectors into sub-vectors, clusters each subspace and stores short codes plus a codebook, reducing memory by up to 90 percent with some recall loss. The standard production pattern is a two-stage search: retrieve a wide candidate set from the compressed index, then re-rank the top candidates against full-precision vectors held on disk or in a smaller cache. That recovers most of the lost recall at a modest latency cost.

Sizing by corpus scale

The threshold that matters most is around 100 million vectors, where in-memory approaches stop being economical and you choose between horizontal sharding and a disk-resident index. Disk-based indexes such as DiskANN, supported natively in recent Milvus releases, keep the graph on NVMe and cache hot layers in memory, which is the practical path for billion-scale corpora that cannot fit in RAM or GPU memory.

Where GPUs actually help

Two distinct jobs get confused here. Index construction is embarrassingly parallel and benefits enormously from GPU acceleration — rebuilding a large index after a model change is otherwise a multi-day CPU job. Query serving is a different question: GPU ANN methods such as CAGRA deliver very high query throughput and are typically recommended in the 1 million to 100 million vector range, where the index fits in GPU memory.

Above that, the index no longer fits and you are back to sharding or disk, at which point the GPU’s role narrows to build and to the embedding model itself. For most enterprise RAG deployments the embedding and generation models consume far more GPU than the index does; the sizing method for that side is in RAG storage and retrieval sizing.

Operational realities people miss

Index build time is the most commonly underestimated number. A billion-vector HNSW build is measured in days on CPU, and it recurs every time you change the embedding model. Budget both the wall-clock and the temporary storage for a parallel build alongside the live index, because you cannot take retrieval offline for a week.

Deletion and update behaviour is the second. HNSW does not delete gracefully; most systems tombstone and periodically compact, which means capacity headroom for a corpus with high churn. Third, replicas multiply everything — a highly available deployment needs at least two copies of whatever you sized, plus room to build the next version. Fourth, storage class matters: a disk-based index on NVMe behaves very differently from one on SATA SSD, and the capacity planning discipline in GPU storage planning for checkpoints and RAG indexes applies directly.

A practical sizing method

Work in this order. Fix the embedding dimension first, since it scales everything. Compute raw bytes, then apply the quantization you intend to use, then add graph overhead, then multiply by replicas, then add roughly 30 percent headroom for growth and compaction. Validate recall at your target k on a held-out query set from your own domain before committing — published benchmark recall on open datasets is a poor predictor for specialised corpora.

Finally, decide sharding on operational grounds as much as capacity. A single large node is simpler to run but slower to rebuild and riskier to lose; three medium shards rebuild in parallel and degrade gracefully. For Indian deployments where the retrieval tier must stay in-country for DPDP reasons, sharding also gives a cleaner story for capacity growth without renegotiating a single large machine, as discussed in the RAG GPU server reference architecture.

Frequently asked questions