HBM3e & GPU Memory: Why It Decides Your Model Size

A GPU's high-bandwidth memory (HBM) sets two things: capacity decides which models fit, and bandwidth decides how fast they run. Because large language models are memory-bound — their speed is limited by moving weights and cache, not by raw math — memory is usually the deciding spec, not FLOPS. In 2026 that means H100 (80 GB HBM3, 3.35 TB/s), H200 (141 GB HBM3e, 4.8 TB/s), and B200 (≈192 GB HBM3e, ~8 TB/s).

TL;DR — key takeaways

• Capacity = what fits. Weights ≈ 2 bytes/param (FP16); a 70B needs ~140 GB, a 405B ~810–970 GB. Memory, not FLOPS, is the gate.
• Bandwidth = how fast. LLMs are memory-bound — tokens/second tracks memory bandwidth more than compute.
• 2026 line-up: H100 80 GB / 3.35 TB/s · H200 141 GB / 4.8 TB/s · B200 ~192 GB / ~8 TB/s (NVIDIA H200; Introl, 2026).
• More HBM per GPU = fewer GPUs per model — often the cheaper, simpler design.

What HBM is and why it matters

High-Bandwidth Memory (HBM) is stacked DRAM sitting next to the GPU die, connected by a very wide bus. It exists because AI workloads move enormous amounts of data — model weights, activations, and the attention KV cache — and ordinary GPU memory can't feed the compute units fast enough. HBM3e is the current generation, pairing large capacity with multi-terabyte-per-second bandwidth.

Capacity decides what fits

Model weights consume roughly 2 bytes per parameter at FP16 (1 byte FP8, 0.5 byte 4-bit). So capacity is a hard gate: a 70B model ≈ 140 GB (FP16), a 405B ≈ 810–970 GB. A model that doesn't fit in one GPU's HBM must be split across GPUs or quantized to a smaller precision. This is why the H200's 141 GB lets a single GPU hold a 70B model plus KV cache that would spill two 80 GB H100s.

Bandwidth decides how fast

LLM inference is memory-bound: each generated token requires reading the model's weights from HBM, so tokens/second scales with memory bandwidth more than with raw FLOPS (Lyceum, 2026). That's why the H100's 3.35 TB/s beats the older A100's ~2.0 TB/s for the same model, and why B200's ~8 TB/s is a step change. For long contexts (128k), high bandwidth is often the only way to keep interactive latency.

The 2026 line-up

Table 1 — Datacenter GPU memory (2026).

GPU	HBM capacity	Bandwidth	Holds (FP16)
H100	80 GB HBM3	3.35 TB/s	up to ~34B; 70B needs 2
H200	141 GB HBM3e	4.8 TB/s	a 70B with KV headroom
B200	~192 GB HBM3e	~8 TB/s	70B comfortably; big KV/context

Assumptions & scope

Weight-memory rules (bytes/param) plus a KV-cache allowance; real footprints vary with context length and precision. A concept explainer — see the sizing guides for exact configs. Specs current July 2026.

Where RDP GPU Mart fits

RDP GPU Mart configures servers around the memory your models need — H200 (141 GB) for single-GPU 70B work, multi-GPU or B-class for larger — so you buy for capacity + bandwidth, not just core count. India-built, INR-transparent, DPDP-aware. *(Configure by GPU memory or request a quote at RDP GPU Mart.)*

FAQ

Why does GPU memory decide model size? Because weights consume ~2 bytes/param (FP16); the model must fit in HBM or be split/quantized. A 70B needs ~140 GB.

What is HBM3e? The current generation of stacked high-bandwidth memory — large capacity plus multi-TB/s bandwidth — used on H200 (141 GB) and B200 (~192 GB).

Is memory or compute the bottleneck for LLMs? Memory. LLM inference is memory-bound; tokens/second tracks bandwidth more than FLOPS.

H100 vs H200 for a 70B model? The H200's 141 GB holds a 70B on one GPU; two H100s are needed at FP16 — fewer GPUs usually means a simpler, cheaper node.

—

Related

• How Much VRAM for a 70B / 405B LLM (Inference)?
• FP8 / FP4 Explained: Precision, Throughput & Cost Trade-offs
• Reference Architecture: 8× H200 On-Prem AI Training Node

Research log (Rule #1)

1. NVIDIA — H200 datasheet (141 GB HBM3e, 4.8 TB/s). https://www.nvidia.com/en-us/data-center/h200/ 2. Introl (2026) — H100 vs H200 vs B200 (capacity/bandwidth). https://introl.com/blog/h100-vs-h200-vs-b200-choosing-the-right-nvidia-gpus-for-your-ai-workload 3. Lyceum (2026) — A100 vs H100 inference (memory-bound). https://lyceum.technology/magazine/a100-vs-h100-for-llm-inference/ 4. apxml (2026) — model VRAM by size/precision. https://apxml.com/models/llama-3-1-405b 5. Runpod (2026) — B200 specs (192 GB, ~8 TB/s). https://www.runpod.io/articles/guides/nvidia-b200