Knowledge Base ›AI Architectures ›Fine-tuning

How Many GPUs Do You Need to Fine-Tune a 70B LLM On-Prem?

Updated 6 Jul 2026 · 7 min read

Fine-tuning a 70-billion-parameter LLM on-premises needs roughly one GPU for QLoRA, two GPUs for standard LoRA, and eight or more for full FP16 fine-tuning. The deciding variable is the tuning method, not the model: QLoRA fits ~46–65 GB, LoRA ~100–140 GB, and full fine-tuning ~400–600 GB of aggregate GPU memory (2026 figures).

How Many GPUs Do You Need to Fine-Tune a 70B LLM On-Prem?

TL;DR — the decisions

Adapting behaviour/tone, tight budget → QLoRA on 1× NVIDIA H200 (141 GB) or 1× H100 (80 GB). ~80–90% of full-FT quality.
Production-grade domain adaptation → LoRA on 2× H200/H100. ~90–95% of full-FT quality.
Full weight update / new capability → full FP16 fine-tuning on an 8× H200 node (1,128 GB) with ZeRO-3 sharding, scaling to multi-node for larger runs.
Rule of thumb: VRAM ≈ inference footprint × 1.2–1.5 (QLoRA), × 1.5–2 (LoRA), × 3–4 (full FT).
On-prem pays back when GPUs run >50–60% utilization; below that, rent. Data-residency (DPDP) often forces on-prem regardless of TCO.

What this guide sizes — and what it doesn't

This sizes GPU memory and GPU count to fine-tune a dense 70B model (e.g. Llama-3.1-70B class) on-premises, plus the network and power that config implies. It does not cover pre-training from scratch, inference-only serving (see the companion inference sizing guide), or mixture-of-experts models, whose active-parameter math differs. Assumptions are stated in the block at the end.

Inputs you need before sizing

Model size & type: dense 70B here; parameter count drives everything.
Tuning method: full fine-tune, LoRA, or QLoRA — the single biggest lever.
Precision: FP16/BF16 vs 4-bit (NF4) base weights.
Sequence length & batch size: these size the activation and KV memory on top of weights.
Dataset size & time budget: sets how many GPUs you need to finish in the training window.

Methodology — the memory math, walked through

Start from the inference footprint, then apply a method multiplier.

Base weights. At FP16, memory ≈ 2 bytes × parameters. For 70B that is ~140 GB. At 4-bit (NF4) it is ~0.5 byte × params ≈ ~35–43 GB (Lyceum Technology, 2026).

Full fine-tuning (FP16 + Adam). Each parameter carries weights, gradients, and optimizer state. The standard mixed-precision Adam budget is ~16 bytes/parameter (2 B weight + 2 B gradient + 8 B optimizer state + 4 B FP32 master copy). For 70B that is ~1.12 TB of model-state memory in theory, before activations. In practice, ZeRO-3 sharding and optimizer offload bring the *achievable* requirement to ~400–600 GB of aggregate GPU memory — about eleven H100 (80 GB) or eight H200 (141 GB) (Spheron, 2026).

LoRA. Base weights stay frozen in FP16 (~140 GB); only small low-rank adapters train, so optimizer state is tiny. Aggregate need is ~100–140 GB → one-to-two H200/H100 (Spheron, 2026).

QLoRA. Base weights are quantized to 4-bit (~35–46 GB) and only adapters train. Aggregate need is ~46–65 GB → a single 80 GB H100 or a single 141 GB H200 with headroom (DEV/QLoRA guide, 2026). QLoRA reaches ~80–90% of full-fine-tune quality; LoRA ~90–95% (Spheron, 2026).

Worked example — Llama-3.1-70B, 4,096-token context

*Figure 1 — On-prem 8× GPU fine-tuning node (existing Higgsfield House-Look asset).*

Method	Aggregate GPU memory	Example on-prem config	Network	GPU power (approx)	Quality vs full FT
QLoRA	~46–65 GB	1× H200 (141 GB) or 1× H100 (80 GB)	single node	~0.7 kW (1 GPU)	~80–90%
LoRA	~100–140 GB	2× H200/H100, NVLink	single node, NVLink	~1.4 kW (2 GPUs)	~90–95%
Full FT (FP16)	~400–600 GB achievable (~1.12 TB theoretical)	8× H200 (1,128 GB), ZeRO-3	NVLink in-node + NDR InfiniBand for multi-node	~5.6 kW GPUs (~10 kW node)	100% (reference)

Table 1 — Fine-tuning a 70B model on-prem: memory, config, network and power by method (2026).

Reading Table 1: most teams do not need a cluster. A domain-adapted 70B assistant is a single- or dual-GPU job via QLoRA/LoRA. You only reach an 8-GPU node when you must update *all* weights.

Why memory bandwidth, not just capacity, sets your throughput

Two GPUs can both "fit" a job and still train at very different speeds. The H100 delivers 80 GB at 3.35 TB/s; the H200 delivers 141 GB at 4.8 TB/s — a memory upgrade on the same Hopper die; the B200 (Blackwell) delivers ~180–192 GB at ~8 TB/s, roughly 2.4× the H100's bandwidth, plus FP4 and NVLink 5.0 (NVIDIA H200 datasheet; Introl, 2026). For fine-tuning, higher bandwidth shortens each step and lets a single H200 absorb a QLoRA job that would need two H100s, which is often the cheaper, simpler on-prem choice.

Scaling & bottleneck notes

Memory-bound below the node, network-bound above it. Inside one server, NVLink keeps GPUs fed. Across servers for full FT, the fabric matters: use NDR InfiniBand (400 Gbps/direction, ConnectX-7) or an equivalent lossless Ethernet fabric, or gradient sync will stall the GPUs.
Storage is the silent bottleneck. Training stalls on data loading far more often than on GPU FLOPS; size an all-flash/parallel tier that can keep 8 GPUs saturated (see the companion storage architecture guide).
Trade capacity for count. One 141 GB H200 can replace two 80 GB H100s for memory-bound jobs — fewer GPUs, less network, lower failure surface.

Assumptions & scope (validated boundary)

Dense 70B model; FP16/BF16 unless 4-bit noted; Adam optimizer; 4,096-token context; ~50–60% model-FLOP utilization typical; figures are *model-state + typical activation* estimates, not a specific tested run — validate against your framework (DeepSpeed/FSDP) and dataset. Multi-node numbers assume ZeRO-3 with a non-blocking fabric. Specs current as of July 2026; GPU specs change — re-check the vendor datasheet.

On-prem, sovereign, and the India build

For Indian CIOs and CTOs, the sizing answer often collides with a data-residency requirement: fine-tuning on proprietary or regulated data (health, BFSI, government) under the DPDP Act frequently means the data — and therefore the GPUs — must stay in-country and in your control. That is the case on-prem serves that rented cloud cannot.

RDP GPU Mart builds these exact configurations as India-manufactured systems: a single-GPU H200 workstation for QLoRA, a 2× H200 server for LoRA, and an 8× H200 GPU server (the DRACO class) for full fine-tuning — designed here, manufactured here, and supported here, with INR-transparent pricing and DPDP-aware, in-country deployment. That keeps the training data sovereign while matching the memory math above. *(Configure or request a quote on the matching GPU server at RDP GPU Mart.)*

FAQ

Can I fine-tune a 70B model on a single GPU? Yes — with QLoRA, a 70B model fits on one 80 GB H100 or one 141 GB H200 (~46–65 GB used), reaching ~80–90% of full-fine-tune quality (DEV, 2026).

How much VRAM does full fine-tuning a 70B model need? About 400–600 GB of aggregate GPU memory in practice (≈1.12 TB of model state in theory), i.e. roughly 8× H200 or 11× H100 with ZeRO-3 sharding (Spheron, 2026).

H100 or H200 for on-prem fine-tuning? Prefer the H200 when a job is memory-bound: its 141 GB at 4.8 TB/s lets one GPU do what two 80 GB H100s would, simplifying the node (NVIDIA H200 datasheet).

Is on-prem cheaper than cloud for this? On-prem typically wins above ~50–60% sustained GPU utilization; below that, rent. Data-residency requirements can make on-prem mandatory regardless of TCO.

What network do I need for multi-GPU full fine-tuning? NVLink inside the server; NDR InfiniBand (400 Gbps/dir) or a lossless Ethernet fabric across servers, or gradient synchronization becomes the bottleneck.

—

JSON-LD (to embed at publish — TechArticle + FAQPage + BreadcrumbList; verify against visible text)

TechArticle: headline, author (+credentials), datePublished 2026-07-04, dateModified, publisher (RDP GPU Mart), proficiencyLevel Expert, about "LLM fine-tuning GPU sizing".
FAQPage: the five Q&As above, verbatim.
BreadcrumbList: Home → AI Architectures → Sizing → this article (must match the visible breadcrumb).

Research log (sources pulled — Rule #1)

1. NVIDIA — H200 datasheet (141 GB HBM3e, 4.8 TB/s). https://www.nvidia.com/en-us/data-center/h200/ 2. Introl (2026) — H100 vs H200 vs B200 comparison (bandwidth, Blackwell). https://introl.com/blog/h100-vs-h200-vs-b200-choosing-the-right-nvidia-gpus-for-your-ai-workload 3. Spheron (2026) — GPU VRAM to fine-tune LLMs: full/LoRA/QLoRA sizing + quality %. https://www.spheron.network/blog/gpu-vram-requirements-fine-tune-llm-2026/ 4. DEV / QLoRA-LoRA 2026 guide — single-GPU 70B QLoRA. https://dev.to/jangwook_kim_e31e7291ad98/fine-tune-llms-with-lora-and-qlora-2026-guide-33lf 5. Lyceum Technology (2026) — VRAM for a 70B model (inference vs training). https://lyceum.technology/magazine/how-much-vram-for-70b-model/

NextBest On-Prem Setup for a Startup Training Small LLMs (<13B)

Ready to deploy?

Talk to an RDP architect about power, cooling and lead time.

Request a Quote