Storage and Data-Path Validation: Proving Storage Will Not Starve Your GPUs
Overview
Storage validation answers one question before production: when every GPU pulls data and the job checkpoints at the same time, does anything stall? A cluster can pass compute and fabric acceptance and still deliver poor training throughput because the data path – local NVMe, the shared filesystem, the storage network – was never tested under GPU-shaped load. Commissioning practice is to benchmark each tier with fio, verify GPUDirect Storage where used, run a checkpoint-shaped burst write test, and finish with a combined soak where a real training job and checkpointing run concurrently while you watch GPU utilisation for dips.
Key takeaways
- Test the tiers separately, then together: local NVMe per node, the shared filesystem at 1/4/8/all clients, then a real training job with concurrent checkpointing – each stage localises a different bottleneck.
- Plan read bandwidth against workload: token-based LLM training is light per GPU, but vision, video and multimodal pipelines are commonly planned at up to 2-4 GB/s per GPU of sustained read, per published storage-sizing guidance.
- Checkpoint writes are bursts, not streams: size the burst from your model (parameters plus optimizer state) and validate the write completes inside your target stall window at full client concurrency.
- Metadata matters as much as bandwidth: dataloaders opening millions of small files stress create/stat/open rates – test with fio small-file and metadata-heavy patterns, not just 1 MB sequential reads.
- The pass signal is GPU utilisation: during the combined soak, sustained dips in GPU busy time aligned with data loading or checkpoints are a storage problem no fio pass can excuse.
Set targets before you test
Numbers without targets are theatre. Derive three targets from your actual workload. Read bandwidth: LLM token training reads comparatively little (tokenised shards are compact), while image, video and multimodal training is routinely planned around multiple GB/s per GPU – published guidance from parallel-filesystem vendors such as WEKA cites up to about 4 GB/s per GPU for demanding vision pipelines. Checkpoint burst: checkpoint size is roughly parameter count times bytes per parameter plus optimizer state – for large models this lands in the hundreds of GB to TB range per full checkpoint; divide by your acceptable stall window (or use asynchronous checkpointing and divide by the drain interval) to get required aggregate write bandwidth. Metadata rate: estimate file opens per second at target dataloader concurrency. Our deeper treatments – storage architecture for AI training and checkpoint storage sizing – cover the sizing math; this article is about proving the built system meets it.
Tier 1: local NVMe, node by node
Run fio on every node’s local NVMe (scratch and cache tiers): sequential read and write at large block size and high queue depth for bandwidth, then 4K random read for IOPS, then a sustained 30-60 minute write to expose thermal throttling and SLC-cache exhaustion – a drive that benchmarks at 7 GB/s for thirty seconds and 1.5 GB/s thereafter is a different drive under training load. Compare every node against the fleet median: a single slow drive (firmware, thermal pad, PCIe lane negotiation) shows up as an outlier immediately. Record per-drive results next to serial numbers in the commissioning record; NVMe with latency spikes during burn-in is one of the defects operators explicitly screen for.
Tier 2: the shared filesystem at scale
Benchmark the shared tier (parallel filesystem, NFS, or object) from 1, 4, 8 and all client nodes concurrently, measuring aggregate and per-client bandwidth. Three signatures to catch: aggregate that stops scaling long before the storage vendor’s quoted number (network bottleneck – check LACP hashing, MTU, and whether storage traffic shares links with the compute fabric); per-client fairness collapsing at high concurrency (one node starving); and metadata cliffs – run small-file read and stat-heavy fio/mdtest-style patterns, because dataloaders over millions of JPEG or parquet files are metadata workloads. If the design uses GPUDirect Storage, validate it explicitly with gdsio and confirm in logs that the DMA path is active rather than silently falling back through the CPU bounce buffer – configuration details are in our GPUDirect Storage and DPU offload guide. On RoCE/InfiniBand-attached storage, re-check the network counters after these runs exactly as you did for the compute fabric.
Tier 3: checkpoint burst testing
Simulate the real event: from all training nodes simultaneously, write checkpoint-sized files (matching your framework’s sharding pattern – one shard per rank for FSDP/DCP-style checkpoints) and time the burst end-to-end. Pass criteria: the burst completes inside your stall budget; no client errors or timeouts; and – often missed – the filesystem recovers to normal read latency immediately after, because a checkpoint that poisons read performance for the next ten minutes still costs goodput. Then test restore: reading the full checkpoint back to all ranks is your recovery-time bound after a failure, and it is usually slower than the write. If you use asynchronous checkpointing, validate that the background drain does not degrade concurrent training reads.
The combined soak: GPUs and storage together
Finish with 12-24 hours of a real training job with real data and production checkpoint cadence, watching per-GPU utilisation (DCGM), dataloader wait time, and storage telemetry on one dashboard. The acceptance signal is simple: no sustained GPU-utilisation dips correlated with data loading or checkpoints. This is the test that catches what synthetic layers cannot – cache misconfiguration, dataloader worker starvation, a shared network link, or a prefetch depth that worked in fio but not in PyTorch. Run it inside the wider burn-in campaign and file the results in the acceptance evidence pack.
Storage validation test matrix
| Test | Tool / pattern | Pass criterion | Typical fault found |
|---|---|---|---|
| Local NVMe bandwidth | fio seq R/W, 1M blocks, QD32, 30-60 min | Fleet-consistent; no throttle collapse | Thermal throttling, slow outlier drive |
| Local NVMe IOPS | fio 4K random read | Per drive spec within tolerance | Firmware, lane negotiation |
| Shared FS scaling | fio from 1/4/8/N clients | Aggregate scales to quoted figure; fair per-client share | Network bottleneck, LACP imbalance |
| Metadata / small files | mdtest or fio small-file mix | Meets dataloader open/stat rate target | Metadata server undersizing |
| GPUDirect Storage | gdsio read to GPU memory | GDS path confirmed active, target GB/s | Silent CPU-path fallback |
| Checkpoint burst | All-rank concurrent write + restore read | Inside stall budget; clean recovery | Write cliff at concurrency, slow restore |
| Combined soak | Real job + checkpoints, 12-24 h | No GPU-util dips tied to storage | Dataloader/cache/prefetch issues |
Frequently asked questions
What per-GPU read bandwidth should I actually specify?
It is workload-dependent: tokenised LLM pre-training often needs well under 1 GB/s per GPU, while high-resolution vision and video pipelines are planned at 2-4 GB/s per GPU in published vendor guidance. Specify against your dominant workload with headroom, and validate the aggregate at full client concurrency rather than trusting a single-client number.
Is fio enough, or do I need MLPerf Storage?
fio (plus mdtest for metadata and gdsio for GDS) covers commissioning well because you control the patterns and can compare against contract numbers. MLPerf Storage is valuable when you want a workload-realistic, comparable benchmark across vendors – useful at procurement time. Neither replaces the combined soak with your real training job.
How big is a checkpoint, roughly?
A useful rule of thumb for mixed-precision training: parameters in bf16 (2 bytes each) plus optimizer state (commonly an additional 12 bytes per parameter for Adam-style optimizers in fp32) – order of 1 TB total state for a 70B-class model before sharding and any optimizer offload. Your framework’s actual checkpoint layout governs; measure one real checkpoint early and size from that.
Should checkpoints go to the same filesystem as training data?
They can, if the system is sized for concurrent read plus burst write – which is exactly what the combined soak verifies. Many designs separate tiers: a fast flash tier or local NVMe for checkpoint landing with asynchronous drain to cheaper capacity or object storage. Whatever the design, test the interaction, not the tiers in isolation.
GPU utilisation dips during data loading – storage or dataloader?
Check storage telemetry at the moment of the dip. If storage shows headroom (low latency, bandwidth below limits), the bottleneck is host-side: too few dataloader workers, Python preprocessing, or prefetch depth. If storage latency spikes concurrently, it is the data path. The combined soak with both dashboards side by side settles this argument in minutes.
Ready to deploy?
Talk to an RDP architect about power, cooling and lead time.