FP8 to FP4: How Low-Precision Training Reshapes Cluster Sizing
Overview
Numeric precision quietly became a first-order cluster-sizing input for 2026–28 training plans. FP8 pretraining is now mainstream practice, and 4-bit training is arriving on its heels: NVIDIA’s NVFP4 recipe — block-microscaled FP4 with techniques like stochastic rounding and Hadamard transforms — has been validated in multi-trillion-token pretraining to 120B-parameter scale with accuracy matching FP8 baselines, and Blackwell-generation tensor cores execute it natively at 2–3× FP8’s arithmetic rate with half the memory. For a buyer, the practical translation: the same model, trained with a modern precision recipe on current silicon, needs a materially smaller (or faster) cluster than a BF16-era plan assumed — and hardware without native low-precision support depreciates accordingly.
Key takeaways
- FP8 is the 2026 default for large-scale pretraining; NVFP4-class 4-bit recipes are validated at scale and moving from research to production.
- Reported gains: up to ~1.6× end-to-end throughput versus BF16 configurations and ~2–3× arithmetic over FP8, with memory halved — larger micro-batches and fewer nodes for the same tokens.
- Low-precision training is a recipe, not a switch: micro-block scaling, stochastic rounding and selective high-precision layers are what preserve convergence.
- Sizing consequence: quote clusters in tokens-per-day at the precision you will actually run, not in peak BF16 FLOPS.
- Procurement consequence: native FP4/FP8 tensor-core support (Blackwell-class onward) is now a depreciation hedge, not a spec-sheet nicety.
Where the savings physically come from
Halving bits helps three ways at once. Compute: tensor cores execute twice the operations per cycle at each halving — Blackwell’s fifth-generation cores run FP4 at twice FP8 rates. Memory: weights, activations and gradients stored in lower precision free capacity for larger micro-batches, which itself improves utilisation; NVIDIA’s JAX/MaxText NVFP4 results report up to 1.59× throughput over BF16 partly from exactly this. Bandwidth and fabric: smaller tensors move faster through HBM and across the cluster network, easing the interconnect pressure described in our cluster networking guide. The compounding is why per-generation “10×” training claims are part silicon, part precision arithmetic.
The recipe, and why it isn’t free
Naive 4-bit training diverges. What makes NVFP4-class training work is a stack of guardrails: 16-element micro-block scaling with E4M3 scale factors, stochastic rounding for unbiased gradients, Hadamard transforms to spread outliers, and selective retention of sensitive operations (first/last layers, certain optimizer states) at higher precision. Two buyer-relevant consequences. Software maturity matters: these recipes ship through frameworks (Transformer Engine, MaxText-class integrations), so a cluster’s usable precision is set by its software stack as much as its silicon. And validation remains workload-specific: published results cover dense and MoE Transformers to ~120B at multi-trillion tokens — teams training unusual architectures should budget convergence-risk experiments before committing a full run at FP4.
Folding precision into cluster sizing
The sizing chain: required tokens × model FLOPs-per-token ÷ (per-GPU sustained FLOPS at your training precision × utilisation) = GPU-days. Running that chain at BF16 rates when you will train at FP8 overstates the cluster by roughly 2×; at NVFP4, more. Worked example: a fine-tuning-scale organisation planning a 30B-model continued-pretrain of 1T tokens sizes to roughly half the nodes on an FP8 recipe versus its BF16 spreadsheet — often the difference between a DRACO-class rack and two. The caveats to keep in the model: sustained ≠ peak (use measured MFU, typically 35–55%), precision gains partially trade against the recipe’s overheads (Hadamard, scaling), and data pipeline and storage must keep up at the higher step rate — see storage planning. The generational roadmap context lives in our 2026–2028 cluster roadmap.
Procurement: precision as a depreciation hedge
Hopper-class hardware trains at FP8 but lacks native FP4; Blackwell-generation parts execute NVFP4 natively; successors extend the pattern. As 4-bit recipes standardise over 2026–28, the effective throughput gap between generations widens beyond the headline FLOPS ratio — which affects both new purchases (native low-precision support should be a hard requirement for training estates) and the second life of existing fleets (FP8-only silicon remains excellent for fine-tuning and inference duty, per the role-change lifecycle in our flagship timing article). For Indian buyers weighing rented IndiaAI-pool capacity against owned nodes, the same adjustment applies to rental math: an hour on FP4-capable silicon buys more training than the per-hour price alone suggests.
Precision capability by generation
| Generation | Native training precisions | Practical 2026 role |
|---|---|---|
| Ampere (A100-class) | BF16/TF32 | Legacy training, fine-tuning, inference |
| Hopper (H100/H200) | + FP8 (Transformer Engine) | Mainstream FP8 pretraining and fine-tuning |
| Blackwell / Blackwell Ultra | + FP4/FP6 (5th-gen tensor cores) | NVFP4-recipe pretraining, highest tokens-per-watt |
| Rubin (2026–27) | FP4-first designs, higher rates | Next-cycle training estates |
Frequently asked questions
Is FP4 training production-ready in 2026?
For dense and MoE Transformer pretraining within validated scales (multi-trillion tokens, up to ~120B published), yes with the full recipe and supported frameworks. Novel architectures and extreme scales still warrant pilot convergence runs.
Does low-precision training hurt final model quality?
Published NVFP4 results show loss and downstream accuracy matching FP8 baselines when the recipe is followed. The risk concentrates in skipping guardrails — naive casting to 4-bit does degrade or diverge.
How much smaller can a cluster be at FP8 vs BF16?
Roughly half the GPU-days for the same tokens, before utilisation effects — and NVFP4 extends the gain further on native silicon. Always size from measured MFU at your actual precision, not datasheet peaks.
Does this apply to fine-tuning too?
Partially — QLoRA-style quantised fine-tuning already exploits 4-bit storage, and FP8 compute helps larger fine-tunes. The dramatic cluster-scale savings, though, accrue to pretraining and continued-pretraining workloads.
Should we avoid buying FP8-only hardware now?
For a new training estate, prefer native FP4-capable generations — the effective-throughput gap will widen as recipes standardise. FP8-only fleets remain strong for fine-tuning and inference, so existing investments redeploy rather than strand.
Ready to deploy?
Talk to an RDP architect about power, cooling and lead time.