How Much VRAM Does a 70B / 405B LLM Need for Inference?

For inference, a 70B-parameter model needs ~140 GB of GPU memory at FP16, ~70 GB at FP8, and ~43 GB at 4-bit — so one to two datacenter GPUs. A 405B model needs ~810–970 GB at FP16, ~486 GB at FP8, and ~243 GB at INT4 — a full 8-GPU node, or as few as two H200s when quantized. Add ~15–30% on top for the KV cache at long context.

TL;DR — the decisions

• 70B → fits 1× H200 (141 GB) at FP8/4-bit, or 2× H100 at FP16.
• 405B → an 8× H100/H200 node at FP8; 2× H200 only with INT4 (AWQ) quantization (NVIDIA, 2026).
• Rule of thumb: VRAM ≈ 2 bytes × params (FP16), 1 byte (FP8), 0.5 byte (4-bit) — then add KV cache for your context length and concurrency.
• Quantization is the biggest lever: INT4 cuts a 405B from ~972 GB to ~243 GB with modest quality loss.

What this sizes — and what it doesn't

GPU memory and GPU count to serve (infer) a dense 70B or 405B model. It excludes training/fine-tuning (see the fine-tuning sizing guide) and mixture-of-experts models (active-parameter math differs). Assumptions are listed at the end.

Methodology — the memory math

Weights. Memory ≈ bytes-per-parameter × parameters:

• FP16 = 2 bytes → 70B ≈ 140 GB, 405B ≈ 810–972 GB (apxml, 2026)
• FP8 = 1 byte → 70B ≈ 70 GB, 405B ≈ 486 GB
• 4-bit = 0.5 byte → 70B ≈ 35–43 GB, 405B ≈ 243 GB (apxml, 2026)

KV cache. On top of weights, the attention KV cache grows with context length × batch/concurrency — budget ~15–30% extra for long-context, multi-user serving. NVIDIA's own 405B example: FP16 weights 140 GB + 112 GB cache ≈ 252 GB (4× H100); FP8 cache halves to ~56 GB (DigitalOcean vLLM guide, 2026).

Worked configurations (2026)

Table 1 — Inference memory + GPU config by model and precision.

Model	Precision	Weights VRAM	+ KV cache	Example config
70B	FP16	~140 GB	+15–30%	2× H100 (80 GB) or 1× H200 (141 GB) tight
70B	FP8	~70 GB	+~15%	1× H200 (141 GB) comfortably
70B	4-bit	~43 GB	+~10%	1× H100 (80 GB)
405B	FP16	~810–972 GB	+100+ GB	8× H200 (1,128 GB) / multi-node H100
405B	FP8	~486 GB	+~56 GB	8× H100 (640 GB) tight → node
405B	INT4 (AWQ)	~243 GB	+	2× H200 (NVIDIA, 2026)

Reading Table 1: a 70B assistant is a single-GPU job at FP8/4-bit. A 405B is a node-class job unless you quantize hard — INT4 brings it down to two H200s, the cheapest way to self-host a frontier-size model on-prem.

Why H200 changes 70B serving

The H200's 141 GB at 4.8 TB/s lets one GPU hold a 70B model plus a healthy KV cache that would spill two 80 GB H100s (NVIDIA H200 datasheet). Fewer GPUs per model means simpler nodes, less networking, and lower failure surface — often the cheaper on-prem choice.

Assumptions & scope

Dense models; standard FP16/FP8/4-bit; figures are weight-memory estimates plus a KV-cache allowance — validate against your serving stack (vLLM/TensorRT-LLM), context length, and concurrency. Quantized quality varies by method; test on your eval set. Specs current July 2026.

Where RDP GPU Mart fits

RDP GPU Mart builds the exact inference boxes this math points to — a 1× H200 server for 70B serving, an 8× H200 DRACO node for 405B — India-manufactured, INR-transparent, and DPDP-aware for in-country deployment. *(Configure an inference server or request a quote at RDP GPU Mart.)*

FAQ

How much VRAM to run a 70B model? ~140 GB at FP16, ~70 GB at FP8, ~43 GB at 4-bit — one H200 or two H100s (apxml, 2026).

Can I run Llama 3.1 405B on 2 GPUs? Yes — with INT4 (AWQ) quantization it fits on 2× H200; at FP8 it needs a full 8-GPU node (NVIDIA, 2026).

What's the VRAM rule of thumb? ~2 bytes/param (FP16), 1 byte (FP8), 0.5 byte (4-bit) for weights, then add 15–30% for the KV cache.

Does quantization hurt quality? Modestly — INT4/FP8 typically retain most quality; always validate on your own eval set before production.

—

Related

• How Many GPUs to Fine-Tune a 70B LLM On-Prem?
• HBM3e & GPU Memory: Why It Decides Your Model Size
• FP8 / FP4 Explained: Precision, Throughput & Cost Trade-offs

Research log (Rule #1)

1. apxml (2026) — Llama 3.1 405B VRAM by precision (FP16 972GB, INT4 243GB). https://apxml.com/models/llama-3-1-405b 2. NVIDIA (2026) — 405B on H200 with TensorRT Model Optimizer (INT4 → 2× H200, 1.44× peak). https://developer.nvidia.com/blog/boosting-llama-3-1-405b-performance-by-up-to-44-with-nvidia-tensorrt-model-optimizer-on-nvidia-h200-gpus/ 3. DigitalOcean (2026) — vLLM GPU sizing (weights + KV cache worked example). https://www.digitalocean.com/community/conceptual-articles/vllm-gpu-sizing-configuration-guide 4. NVIDIA — H200 datasheet (141 GB, 4.8 TB/s). https://www.nvidia.com/en-us/data-center/h200/ 5. K8s Recipes — Llama 70B FP16 = 140 GB. https://kubernetes.recipes/recipes/ai/llama-70b-model-size-gpu-requirements/