Local NVMe Scratch vs Shared Storage: Data Locality for GPU Nodes
Overview
The fastest storage in any GPU cluster is the NVMe already inside the node: a pair of Gen5 drives delivers 20+ GB/s and millions of IOPS with no network, no metadata server, and no contention from the job next door. The design question is not local versus shared – every serious cluster needs both – but which data lives where. The working rule: shared storage is the source of truth; local NVMe is a disposable performance layer for staged datasets, shuffle and temporary files, and first-hop checkpoint writes. Teams that formalise this staging pattern routinely feed GPUs from local flash at a fraction of the cost of scaling the shared tier to do the same job.
Key takeaways
- Local NVMe wins on latency (~70-100 us, no network hops), per-node bandwidth, and cost per delivered IOPS; shared storage wins on capacity, durability, and any data more than one node must see.
- Staging – copying the dataset shard to local scratch at job start – converts shared-storage random I/O into one sequential read, then serves the whole job locally.
- Checkpoint-to-local-then-drain cuts GPU stall time: the burst lands on local flash in seconds and drains to shared storage asynchronously.
- Common sizing for 2026 8-GPU nodes: 4-8 x 3.84-7.68 TB NVMe (15-60 TB raw) as RAID-0 or independent scratch – treat it as disposable.
- Local scratch has node-failure semantics: anything not yet drained is gone with the node, so the orchestration layer must know what is a copy and what is an original.
Why local flash is so hard to beat on performance
A local NVMe read traverses PCIe and nothing else. There is no fabric serialisation, no lossless-Ethernet tuning, no metadata server round trip, and – critically for shared clusters – no noisy neighbour: the job owns the drives. Eight Gen5 drives in a node aggregate to roughly 100 GB/s of read bandwidth, more than most shared filesystems deliver to an entire rack. The catch is equally simple: that performance is invisible to every other node, unprotected against node loss, and capped by the chassis slot count. This is why the industry pattern is a hierarchy, not a choice – the same logic that produces the cache tiers described in tiering architecture for AI data.
Pattern 1: explicit staging
The oldest HPC pattern still works best for training. At job start, the scheduler (Slurm prolog, Kubernetes init container, or the framework’s own data pipeline) copies the job’s dataset shard from shared storage to local scratch as large sequential reads – the friendliest possible load for the shared tier. The job then trains entirely against local flash; epochs two through N never touch the network. Staging suits packed datasets (see the small-file and metadata problem for why you should pack anyway) and any workload whose working set fits node-local capacity. The operational work is cleanup discipline: scratch must be swept between jobs, or the fourth user of the node finds full drives.
Pattern 2: transparent cache tiers
Where working sets exceed scratch or datasets change mid-job, cache layers hydrate local NVMe on demand and keep the namespace unified: Alluxio and JuiceFS cache remote object or file data locally; NVIDIA AIStore, DDN Infinia-style client caches, and filesystem-native client caching (WEKA’s, Lustre PCC) do similar work inside their stacks. Reads hit local flash after first access; the platform manages eviction. The trade-offs are cache-miss latency cliffs on cold data and another distributed system to operate – worth it for large multi-tenant estates, overkill for a lab that could just stage. Inference has its own variant of this pattern: KV-cache and model-weight caching on local NVMe, covered in KV cache offloading.
Pattern 3: local-first checkpointing
Checkpoint stalls shrink dramatically when the synchronous write goes to local NVMe and a background process drains it to durable shared storage. Frameworks increasingly support this natively (asynchronous and hierarchical checkpointing in NeMo, DeepSpeed, and PyTorch DCP). The design obligations: the drain must complete before the next checkpoint overwrites scratch, and restart logic must fetch from shared storage when the original node is dead. Sizing that drain bandwidth belongs to your fabric design – see storage network design.
What lives where
| Data | Home | Local NVMe role | On node failure |
|---|---|---|---|
| Training datasets (source of truth) | Shared FS / object store | Staged copy per job | Re-stage on another node |
| Shuffle, augmentation, temp files | Local scratch only | Native home | Recomputed; nothing lost |
| Checkpoints | Shared durable tier | First-hop landing, async drain | Lose only undrained delta |
| Model weights for inference | Registry / object store | Cached copy | Re-pull on reschedule |
| Code, environments, containers | Registry / shared FS | Image cache | Re-pull |
| Experiment results, logs | Shared FS / object store | Buffer at most | Flush frequently |
Sizing scratch for a GPU node
Work from the largest concurrent job’s per-node working set: dataset shard plus shuffle space plus one or two checkpoint generations, times a safety factor for multi-tenancy. For 2026-class 8-GPU nodes, 4-8 drives of 3.84-7.68 TB (TLC, 1-3 DWPD – scratch sees constant writes; drive selection guidance in all-flash NVMe design) is the common band. E1.S slots make 8-16 drives feasible even in dense chassis. Skip parity RAID on scratch: RAID-0 or independent filesystems per drive maximise bandwidth, and the failure answer is re-stage, not rebuild. One India-specific note: local NVMe is also the tier least exposed to import lead times – adding drives to existing nodes is often the fastest capacity you can buy under allocation.
Frequently asked questions
Can I skip shared storage entirely for a small cluster?
For single-node jobs, nearly – an object store or NAS for durable copies plus local NVMe for everything hot is a legitimate minimal design. Multi-node training needs a shared tier for datasets and checkpoints, even a modest one.
Should scratch be RAID-protected?
No. Scratch holds copies and recomputables by definition. RAID-0 or per-drive filesystems give maximum bandwidth; protection budget belongs on the shared tier.
How do I stop users treating scratch as permanent?
Enforce, do not request: per-job scratch directories created and deleted by the scheduler, TTL sweeps, and quotas. Every mature HPC centre converged on automated cleanup because policy documents alone never worked.
Does GPUDirect Storage apply to local drives?
Yes – GDS moves data from local NVMe straight into GPU memory, skipping the CPU bounce buffer, and local drives are its lowest-latency case. See our GPUDirect Storage article for setup implications.
What utilisation should the staged pattern achieve?
After the first epoch, storage should effectively vanish from the profile – dataloader wait near zero because all reads are local. If profiling still shows I/O stalls, the bottleneck has moved to decode/augmentation CPU, not storage.
Ready to deploy?
Talk to an RDP architect about power, cooling and lead time.