Fine-Tuning LLMs on a Workstation: LoRA and QLoRA Memory Math
Overview
Fine-tuning open-weight LLMs up to 70B parameters is now a single-workstation job — provided you use parameter-efficient methods and do the memory arithmetic before buying. On a 96 GB-class card, QLoRA handles 70B (base model in 4-bit at ~35–40 GB, leaving ~55 GB for adapters, optimiser states and activations); LoRA in FP16 covers models to ~32B; and full fine-tuning of anything beyond 7B remains firmly out of reach on one GPU, since optimiser states alone multiply weight memory several times. Sizing references such as Spheron’s 2026 VRAM guide and Modal’s fine-tuning memory analysis converge on the same numbers this article works through.
Key takeaways
- Fine-tuning memory = weights + gradients + optimiser states + activations; PEFT methods shrink the middle two to under 1% of parameters.
- QLoRA rule-of-thumb VRAM: ~12 GB for 7B, ~20 GB for 13B, ~44 GB for 32B, ~88 GB for 70B — the last fitting a 96 GB card with careful settings.
- LoRA at FP16 costs roughly 2× the base-weight footprint; it beats QLoRA slightly on quality/speed when VRAM allows.
- Full fine-tuning needs 8–12× the weight memory (AdamW states + activations) — server territory beyond 7B.
- Sequence length and batch size are the hidden budget: activations scale with both, and long-context fine-tunes can double the plan.
Where the memory actually goes
Four allocations compete for VRAM. Base weights: 2 bytes/parameter at FP16, ~0.5 at 4-bit. Gradients: one per trainable parameter — the reason PEFT works, since LoRA adapters make under 1% of parameters trainable. Optimiser states: AdamW keeps two moments per trainable parameter, so full fine-tuning pays ~12 bytes/parameter here while LoRA pays almost nothing. Activations: proportional to batch size × sequence length × depth, tamed by gradient checkpointing at ~20–30% recompute cost. The design insight of QLoRA is simply that the largest allocation — frozen base weights — tolerates 4-bit quantisation during training without wrecking adapter quality.
The sizing table that matters
Working numbers for common open-weight sizes (Llama/Qwen/Mistral-class), single GPU, moderate sequence lengths: 7B — QLoRA ~12 GB, LoRA ~20–28 GB; 13–14B — QLoRA ~20–22 GB, LoRA ~35–40 GB; 32B — QLoRA ~44 GB, LoRA ~70–80 GB; 70B — QLoRA ~88 GB, LoRA not on one card. Read them against the hardware tiers: a 24 GB card (CARINA-class entry) fine-tunes 7B QLoRA honestly; a 48 GB card covers 13B LoRA and 32B QLoRA; the 96 GB QUASAR tier covers everything to 70B QLoRA. That last configuration — 4-bit base at 35–40 GB plus ~55 GB of headroom — is the practical ceiling of desk-side fine-tuning in 2026 and the reason this card class exists.
Ten gigabytes short: the levers, in order
When a run OOMs, spend cheap levers first. Shorten sequence length to what the task needs — activations drop linearly. Cut micro-batch to 1 and recover throughput with gradient accumulation — near-free. Enable gradient checkpointing — ~25% slower, large savings. Switch optimiser to a paged 8-bit AdamW variant — offloads state pages to system RAM, saving 2–6 GB on smaller models. Reduce LoRA rank (r=64 to r=16 rarely hurts task quality measurably). Last resort, drop base precision from FP16 LoRA to 4-bit QLoRA. If all levers are spent and it still doesn’t fit, the job has outgrown the card — not the tuning; see the escalation section.
When the job needs a server anyway
Three signals: you need full fine-tuning (continued pretraining, major behaviour shifts adapters can’t reach) beyond 7B; dataset scale makes single-card wall-clock unacceptable — a 70B QLoRA epoch over millions of samples runs days on one GPU; or you’re fine-tuning as a team, queueing behind one machine. Then the arithmetic in how many GPUs to fine-tune a 70B LLM and fine-tuning server sizing takes over. The workstation still earns its keep afterwards running evaluation, quantisation and inference on the artefacts — and for Indian teams it keeps proprietary training data on premises, inside DPDP-scoped infrastructure, throughout.
Fine-tuning fit by GPU tier
| GPU tier | QLoRA ceiling | LoRA (FP16) ceiling | Full fine-tune ceiling |
|---|---|---|---|
| 24 GB (entry / CARINA-class) | 13B (tight), 7B comfortable | 7B | — |
| 48 GB (L40S-class) | 32B | 13–14B | ~1–3B |
| 96 GB (QUASAR-class) | 70B | 32B | ~7B (tight) |
| 2× 96 GB tower | 70B faster / longer context | 70B (sharded) | ~13B |
| 8-GPU server (DRACO-class) | — unnecessary | Any open-weight size | 70B+ |
Frequently asked questions
Does QLoRA produce worse models than LoRA?
Published comparisons show near-parity on most task fine-tunes; the 4-bit base costs a small, usually acceptable quality margin. When VRAM allows FP16 LoRA, take it — but never at the cost of truncating your training sequences.
Can a 96 GB card really fine-tune a 70B model?
Yes, via QLoRA: ~35–40 GB for the 4-bit base leaves ~55 GB for adapters, optimiser state and activations at moderate batch and sequence settings. Long-context 70B fine-tunes or FP16 LoRA at 70B need multiple GPUs.
Why is full fine-tuning so much more expensive?
Every parameter becomes trainable, so gradients and AdamW’s two moment tensors apply to all of them — roughly 8–12× the weight memory before activations. That is why PEFT methods dominate below server scale.
What LoRA rank should I start with?
r=16 with alpha=32 is a sound default; move to r=64 only if evaluation shows the task needs more capacity. Rank raises adapter memory and training time linearly for usually marginal gains.
How long does a workstation fine-tune take?
Order of magnitude on one 96 GB card: hours for 7–13B on tens of thousands of samples; a day-plus for 32B; multiple days for 70B QLoRA on large datasets. If wall-clock blocks iteration, that — not VRAM — is the signal to scale out.
Ready to deploy?
Talk to an RDP architect about power, cooling and lead time.