Storage Network Design for AI Clusters: Dedicated vs Converged Fabric
Overview
An AI cluster carries three traffic classes: GPU-to-GPU east-west (gradient exchange, all-reduce), storage (dataset reads and checkpoint bursts), and management. The design question is whether storage rides the same physical fabric as GPU traffic or gets its own network. The short answer: small and inference-focused clusters converge safely with QoS discipline; large training clusters separate, because a multi-terabyte checkpoint burst arriving in the middle of an all-reduce is exactly the collision you built the expensive fabric to avoid. NVIDIA’s own reference architectures publish both patterns – convergence for enterprise-scale, dedicated storage fabrics for large training pods.
Key takeaways
- Reference designs such as NVIDIA’s HGX AI Factory architecture typically give storage two 400 Gb/s ports per node on a fabric separate from the rail-optimised GPU backend.
- Convergence saves switches, optics, and power – meaningful in Indian facilities where rack power and imported optics dominate cost – but demands correct PFC/ECN and DSCP-based traffic classes on every port.
- The failure mode of bad convergence is silent: training slows during checkpoint windows and nobody’s dashboard says why.
- Checkpoint write bursts are the sizing event for the storage fabric, not steady-state dataset reads.
- Never converge storage onto the GPU backend rails in large training pods; if you converge, converge onto the frontend/north-south network.
The three-fabric model
Current GPU server reference architectures wire each node into: (1) a GPU backend fabric – rail-optimised, RDMA (InfiniBand or RoCEv2), one NIC per GPU, reserved for collective operations; (2) a frontend or converged north-south network – typically two 200-400 GbE ports carrying user access, orchestration, and in many designs storage; and (3) out-of-band management. The architectural decision is where storage attaches: its own dedicated pair of ports and switches, the frontend network, or (the option to resist) the GPU backend. Treat the backend as sacred: collective operations are latency- and jitter-sensitive, and every microsecond of queueing there is GPU time across the whole job.
The case for converging storage
Convergence means storage shares the frontend Ethernet with other north-south traffic. The savings are concrete: fewer leaf switches, fewer optics (at 400G, optics are a major line item), fewer cables, less rack power, and one less fabric to operate. Modern Ethernet makes it technically sound: DSCP-marked traffic classes, Priority Flow Control on the storage class, and ECN give storage lossless behaviour while bounding how much bandwidth it can take. For clusters up to roughly 8-16 nodes, for inference and fine-tuning estates, and for RAG serving – workloads whose storage traffic is steady reads rather than synchronized bursts – a converged 2×400 GbE frontend is usually the right call, and it is what several enterprise reference designs now ship.
The case for a dedicated storage fabric
Training at scale changes the traffic shape. Checkpoints are synchronized, cluster-wide write bursts – every node flushing gigabytes in the same window, hundreds of GB/s aggregate at frontier scale (sizing in checkpoint storage at frontier scale). Epoch starts and data-parallel restarts produce synchronized read storms. When these share links with anything else, they either get throttled (checkpoint time stretches, GPUs stall) or they win (user traffic and orchestration suffer). A dedicated storage fabric – typically two 400 GbE ports per node into their own leaf-spine, RoCEv2 end to end – makes storage performance a provisioned quantity instead of a QoS negotiation. It also isolates blast radius: a storage-network incident cannot take orchestration down with it. From roughly 32 GPU nodes upward, or wherever checkpoint bandwidth is contractual (training-as-a-service), dedicated is the defensible default.
Rules that make either design work
- Lossless where RDMA runs: PFC enabled on the storage traffic class, ECN (DCQCN) tuned, and headroom buffers sized per switch vendor guidance. Test with synthetic incast, not just steady streams.
- Classify at the edge: DSCP marking at the host (or DPU), verified on every hop. An unmarked flow silently lands in best-effort.
- Watch oversubscription: storage leafs to spine at 1:1 for dedicated fabrics; converged designs should keep storage-class bandwidth guarantees explicit (e.g. 50 percent minimum on shared links).
- Two ports, two switches: dual-home every node’s storage attachment; single-homed storage turns a switch reboot into a cluster event.
- Monitor the class, not the port: per-traffic-class counters (PFC pause frames, ECN marks, class drops) are the early-warning signals; port-level utilisation hides the story.
Recommendation by cluster size
| Cluster profile | Storage attachment | Transport | Rationale |
|---|---|---|---|
| 1-8 GPU nodes, mixed use | Converged on frontend, 2×100-200 GbE | NVMe/TCP or NFS/RDMA | Traffic too small to justify a fabric |
| 8-32 nodes, fine-tuning/inference | Converged on frontend, 2×400 GbE with QoS | RoCEv2 or TCP | Steady reads; QoS handles contention |
| 32-256 nodes, training pods | Dedicated storage leaf-spine, 2×400 GbE/node | RoCEv2 | Checkpoint bursts need provisioned bandwidth |
| 256+ nodes / AI factory | Dedicated fabric, 1:1 oversubscription | RoCEv2 (or IB storage fabric) | Storage SLA independent of compute fabric |
| Any size, GPU backend rails | No storage here | – | Collectives own this fabric |
Cost notes for Indian deployments
A dedicated storage fabric for a 64-node pod adds roughly four to six 400G-class switches plus 128 optics and their power draw – hardware that competes for the same import budget and rack power as GPUs. Two mitigations work well: run the dedicated storage fabric at 200 GbE where checkpoint maths allows (halving optics cost while keeping isolation), and pull checkpoint frequency into the calculation before buying bandwidth – asynchronous and sharded checkpointing can cut the burst requirement substantially, as discussed in AI factory storage planning. The transport-level trade-offs behind RoCEv2 versus TCP on either fabric are covered in NVMe over Fabrics for AI storage.
Frequently asked questions
Can storage share the InfiniBand compute fabric?
Technically yes, and some HPC sites do. For AI training pods the guidance is no: collective operations are jitter-sensitive, and checkpoint bursts are exactly the traffic you do not want queueing against all-reduce. Keep the backend for GPUs.
Is 2×400 GbE per node overkill for storage?
For steady dataset reads, usually. For checkpoint bursts it sets your floor on checkpoint duration: a node flushing 500 GB over 2×400 GbE takes ~5 seconds at line rate. Work backwards from acceptable checkpoint stall time.
Do I need lossless Ethernet for NVMe/TCP?
No – TCP handles loss itself, which is part of its operational appeal. You still want QoS classes so storage and other traffic do not starve each other on converged links.
What breaks first in a badly converged design?
Nothing visibly. Checkpoint windows stretch, dataloader latency grows tails, and GPU utilisation sags a few percent. Per-traffic-class counters (pause frames, ECN marks) are where the evidence lives.
Does a DPU change this decision?
DPUs (for example BlueField-3) offload storage protocol processing and enforce isolation at the host edge, which strengthens the converged case for mid-size clusters. They do not remove the physics of shared-link contention during synchronized bursts.
Ready to deploy?
Talk to an RDP architect about power, cooling and lead time.