FP8 / FP4 Explained: Precision, Throughput & Cost Trade-offs

Precision is how many bits represent each number in a model. Lowering it from FP16 to FP8 to FP4 roughly halves memory and increases throughput at each step, letting the same GPU hold bigger models and serve more users — at some accuracy cost. FP16 uses ~2 bytes/parameter, FP8 ~1 byte, FP4 ~0.5 byte, so a 70B model shrinks from ~140 GB to ~70 GB to ~35 GB. Blackwell-class GPUs add native FP4 for a large inference throughput gain.

TL;DR — key takeaways

• Memory scales with precision: FP16 ~2 B/param · FP8 ~1 B · FP4 ~0.5 B → a 70B goes 140 → 70 → 35 GB.
• Throughput rises as precision drops — more tokens/second and users per GPU.
• FP8 is the mainstream production sweet spot; FP4 (native on Blackwell) pushes inference density further (NVIDIA GB200).
• Accuracy trade-off is modest with good quantization — but always validate on your eval set.

What precision means

Every weight and activation in a model is a number stored with a fixed number of bits. FP16/BF16 (16-bit) is the standard training/serving precision. FP8 (8-bit) and FP4 (4-bit) are lower-precision formats that store each number in fewer bits — trading numeric detail for smaller size and faster math. Modern GPUs have dedicated tensor-core support for these formats, so the speedup is real, not just theoretical.

The three levers precision moves

Memory (fit). Bytes-per-parameter halve at each step: FP16 ~2 B, FP8 ~1 B, FP4 ~0.5 B. A 70B model needs ~140 GB (FP16), ~70 GB (FP8), ~35 GB (FP4) — the difference between two GPUs and a fraction of one.

Throughput (speed/cost). Lower precision moves less data per token and runs faster tensor-core ops, so tokens/second and concurrent users per GPU rise. Blackwell adds native FP4; NVIDIA quotes large inference gains for GB200 at FP4 (peak/vendor figures) (NVIDIA GB200 NVL72).

Accuracy (quality). Fewer bits can lose precision, but modern methods (per-tensor/channel scaling, AWQ, GPTQ) keep quality close to FP16 for most tasks. INT4/FP4 typically retain most quality; the right answer is measured on your own evals, not assumed.

When to use which

Table 1 — Choosing precision (inference).

Precision	Bytes/param	70B footprint	Use when
FP16/BF16	~2	~140 GB	max fidelity; training; small models
FP8	~1	~70 GB	production serving sweet spot
FP4 / INT4	~0.5	~35 GB	max density; fit a big model on fewer GPUs

Assumptions & scope

Bytes-per-parameter are weight estimates; add KV-cache overhead for serving. Quantized quality varies by method and task — validate before production. A concept explainer; see the sizing guides for configs.

Where RDP GPU Mart fits

RDP GPU Mart sizes servers to the precision you'll run — FP8 on an H200 to serve a 70B on one GPU, or FP4-capable B-class systems for maximum inference density — India-built, INR-transparent, DPDP-aware. *(Configure for your precision or request a quote at RDP GPU Mart.)*

FAQ

What is FP8 / FP4? 8-bit and 4-bit numeric formats for model weights/activations — lower precision than FP16, using ~1 byte and ~0.5 byte per parameter respectively.

How much memory does FP8 save? About half versus FP16 — a 70B model drops from ~140 GB to ~70 GB, often fitting one GPU instead of two.

Does lower precision hurt accuracy? Modestly with good quantization (AWQ/GPTQ) — usually close to FP16. Validate on your eval set before production.

Which GPUs support FP4? Blackwell-class GPUs (e.g. B200, GB200) add native FP4 for high-density inference.

—

Related

• HBM3e & GPU Memory: Why It Decides Your Model Size
• How Much VRAM for a 70B / 405B LLM (Inference)?
• What Is an AI Factory? Rack-Scale AI Explained

Research log (Rule #1)

1. NVIDIA — GB200 NVL72 (native FP4, inference gains — peak/vendor). https://www.nvidia.com/en-us/data-center/gb200-nvl72/ 2. apxml (2026) — 405B VRAM by precision (FP16/FP8/INT4). https://apxml.com/models/llama-3-1-405b 3. NVIDIA (2026) — 405B on H200 with INT4 (TensorRT Model Optimizer). https://developer.nvidia.com/blog/boosting-llama-3-1-405b-performance-by-up-to-44-with-nvidia-tensorrt-model-optimizer-on-nvidia-h200-gpus/ 4. Introl (2026) — Blackwell FP4 / new tensor cores. https://introl.com/blog/h100-vs-h200-vs-b200-choosing-the-right-nvidia-gpus-for-your-ai-workload 5. Spheron (2026) — quantization quality trade-offs. https://www.spheron.network/blog/gpu-vram-requirements-fine-tune-llm-2026/