Sizing On-Prem AI by Model Class: From 3B Laptops to 671B Servers
Overview
The most common procurement mistake in on-premises AI is choosing hardware by GPU brand or budget and discovering afterwards which models it can actually run. The reliable method is the reverse: start from the model class you intend to run and fine-tune, then size the machine to it. This article works through a tiered reference range — laptop, PC, workstation and server — showing what each tier targets, why memory arithmetic decides the tier, and how NVIDIA and AMD paths compare at each step.
Key takeaways
- Size by model class, not budget — the model you must fine-tune determines the tier, everything else follows.
- Laptop and PC tiers are for development — 3B to 13B, QLoRA/LoRA work and edge testing.
- Workstations cover 13B to 180B with per-card VRAM and NVMe offload, no data-centre required.
- Servers start where NVLink/NVSwitch does — 8-GPU nodes pooling 640 GB to 1536 GB for 180B to 671B.
- AMD is a real alternative at the top — MI300X/MI325X offer higher VRAM per GPU via xGMI where ROCm suits the stack.
The arithmetic that decides your tier
Model memory is parameters multiplied by bytes per parameter: 4 bytes at FP32, 2 at FP16. So 13B at FP32 is roughly 52 GB, 34B about 136 GB, 70B about 280 GB, 180B about 720 GB. Fine-tuning adds optimiser state and gradients on top of weights, which is why a machine that serves a model comfortably may be unable to train it — the distinction that catches most buyers out. Decide first whether you need inference only or fine-tuning, because the same model sits in different tiers depending on the answer.
The tiers
| Tier | Target model | NVIDIA path | AMD alternative | Typical capability |
|---|---|---|---|---|
| Laptop (iGPU) | 3B | Intel Arc iGPU, 96 GB LPDDR5X | Radeon 890M iGPU | Edge 3B inference + QLoRA dev |
| Laptop (dGPU) | 7B | RTX 4070 Laptop 8 GB | RX 7700M 8 GB | 7B QLoRA/LoRA + BF16 inference |
| PC | 13B | RTX 4080 Super 16 GB | RX 7900 XTX 24 GB | 13B QLoRA/LoRA fine-tune |
| Workstation | 34B-90B | RTX A4000/A6000 BW, 2x 48-96 GB | Radeon Pro W7900 / MI300X | Professional fine-tuning, ECC |
| Workstation (top) | 180B | 4x RTX A6000 BW 96 GB = 384 GB | 2x MI300X 192 GB via xGMI | 180B with offload |
| Server | 180B-405B | 8x H100/H200, NVSwitch 640 GB-1128 GB | 8x MI300X/MI325X, 1536-2304 GB | Large training, high concurrency |
| Server (flagship) | 671B | 8x B200 192 GB, 1536 GB + FP8 | 8x MI325X 288 GB, 2304 GB | Frontier-class on-prem |
Where the architectural break happens
The meaningful discontinuity is not price — it is GPU-to-GPU interconnect. Workstation tiers use multiple cards on a per-card basis with no NVLink or NVSwitch, so each GPU’s memory is largely its own and model parallelism is constrained. Server tiers introduce NVSwitch, pooling eight GPUs into a single coherent memory domain: 640 GB on H100 SXM5, 1128 GB on H200, 1536 GB on B200. That pooling is what makes very large models practical rather than merely possible. If your target model needs a unified memory space larger than one card, you are buying a server, whatever the budget suggests — the same logic as our rack-scale AI factory analysis at the next scale up.
Choosing the NVIDIA or AMD path
Both paths are legitimate and the trade is consistent. NVIDIA is the primary route for a mature CUDA software ecosystem — the safe default when your stack, libraries and team experience assume it. AMD offers notably higher VRAM per GPU at the top tiers: MI300X at 192 GB and MI325X at 288 GB, pooled over xGMI to 1536 GB or 2304 GB in an eight-GPU node, which the source characterises as roughly 2.4x an H100 node and 2x an H200 node respectively. Where ROCm supports your framework and VRAM capacity is the binding constraint, that is a serious advantage. Storage and system design remain common to both.
Why this shape suits Indian deployments
A tiered on-premises range answers two local constraints at once. Power and siting: workstation tiers run on office power without data-centre cooling, so a team can start doing real fine-tuning without a facility project, and only step up to server tiers when model scale genuinely demands it. Data control: every tier keeps training data behind your own firewall, which under the DPDP Act — penalties up to ₹250 crore per violation — is the simplest residency position for regulated BFSI, healthcare and public-sector work. Where offload extends a tier to reach a larger model, treat that as capacity rather than speed and validate performance in a scoped proof-of-concept, as covered in our fine-tuning capacity analysis.
Frequently asked questions
How do I know which tier I need?
Start from the largest model you must fine-tune, not run. Multiply parameters by 4 bytes for FP32 or 2 for FP16 to get weights, then add optimiser state and gradients for training. That figure, not your budget, determines the tier.
What is the difference between a workstation and a server tier?
Interconnect. Workstation tiers use multiple cards without NVLink or NVSwitch, so memory is largely per-card. Server tiers pool eight GPUs over NVSwitch into one coherent domain — 640 GB on H100, 1128 GB on H200, 1536 GB on B200.
Can a laptop really fine-tune a model?
Yes, within limits — the laptop tiers target 3B and 7B with QLoRA/LoRA parameter-efficient methods plus BF16 inference. They are development and edge-testing machines, not production training hardware.
When is AMD the better choice?
When VRAM capacity per GPU is the binding constraint and ROCm supports your framework. MI300X at 192 GB and MI325X at 288 GB pool via xGMI to 1536 GB and 2304 GB in eight-GPU nodes, above the equivalent NVIDIA node capacities.
Does ECC memory matter?
It is standard on workstation and server tiers and worth having for long training runs, where a silent memory error can corrupt a job that has been running for days. Laptop and PC tiers use non-ECC memory.
Ready to deploy?
Talk to an RDP architect about power, cooling and lead time.