AI Fabric Fundamentals: Scale-Up, Scale-Out and Why Networks Decide GPU Utilisation
Overview
Buy eight GPUs and you have a server. Buy eighty and you have a distributed systems problem. In a GPU cluster the network is not a utility that connects the compute — it is part of the compute, because every training step ends with GPUs exchanging gradients and waiting for each other. Get the fabric wrong and expensive accelerators sit idle at maybe 40% utilisation while the network catches up. This article explains the two domains that make up an AI fabric, why collective operations punish the slowest path, and what actually determines whether you use the hardware you paid for.
Key takeaways
- Two fabrics, not one — scale-up inside the node or rack, scale-out between nodes, with very different bandwidths.
- Collectives run at the speed of the slowest participant, so tail latency and jitter matter more than average throughput.
- AI fabrics must behave losslessly — a dropped packet triggers retransmission that stalls an entire collective.
- Topology beats raw speed — rail-optimised designs give each GPU a dedicated path to its peers.
- Utilisation is the real KPI — a cheaper fabric that idles GPUs is the most expensive option available.
Two domains: scale-up and scale-out
Every AI cluster has two distinct networks. Scale-up is the tightly coupled domain inside a node or rack — NVLink and NVSwitch in NVIDIA systems, xGMI in AMD — where GPUs share memory coherently at bandwidths measured in terabytes per second. Scale-out connects those nodes to each other over InfiniBand or Ethernet, typically at hundreds of gigabits per second per port. The gap between them is roughly an order of magnitude, which is why the boundary between the two is the most important line in your architecture: work that fits inside the scale-up domain runs fast, and work that crosses the boundary runs at fabric speed. Our note on designing past one rack covers where that line falls in practice.
Why collectives punish the slowest link
Distributed training is not a stream of independent messages. It is dominated by collective operations — all-reduce, all-gather, reduce-scatter — in which every GPU contributes and every GPU waits for the result. The consequence is unforgiving: a collective completes only when its slowest participant completes. One congested link, one mis-cabled port, one node with a degraded transceiver, and the entire job runs at that link’s pace. This is why AI fabric engineering obsesses over tail latency and jitter rather than average bandwidth, and why a fabric that benchmarks well on iperf can still deliver poor training throughput.
Lossless behaviour and congestion control
Ordinary datacentre networks tolerate packet loss because TCP retransmits and users never notice. AI fabrics cannot. GPU communication libraries use RDMA, where a dropped packet forces retransmission that stalls a collective and idles every GPU in the group. AI fabrics therefore need lossless or near-lossless behaviour, delivered through priority flow control and explicit congestion notification, plus congestion-control algorithms tuned for the bursty, synchronised traffic that collectives produce. This is the substantive difference between InfiniBand, which was designed for this, and Ethernet, which has been extended to do it — the trade-off examined in our InfiniBand versus Spectrum-X versus Ethernet comparison.
Rail-optimised topology
The dominant pattern for scale-out AI is the rail-optimised design. Rather than treating every NIC as generic, each GPU in a node is assigned to a numbered rail, and all GPUs with the same index across all nodes connect to the same leaf switch. GPU 0 in every server talks to GPU 0 everywhere else over rail 0. Because collectives typically exchange data between same-index GPUs, most traffic stays within a rail and never traverses the spine at all, cutting hop count and contention. It also makes the failure domain legible: losing a rail degrades the cluster predictably instead of randomly.
Sizing the fabric
| Design question | What to specify | Failure mode if ignored |
|---|---|---|
| Scale-up domain size | GPUs sharing coherent memory (node or rack) | Model sharding spills to slow fabric |
| Per-GPU scale-out bandwidth | NIC port speed per GPU, not per node | Node becomes the bottleneck |
| Oversubscription | Ideally 1:1 non-blocking leaf to spine | Congestion during all-reduce |
| Topology | Rail-optimised leaf assignment | Unnecessary spine traversal |
| Loss behaviour | PFC / ECN configured and tested | Retransmits stall collectives |
| Cabling and optics | Validated at target rate, labelled | Silent degradation, hard to trace |
What this means commercially
The fabric is typically a modest share of cluster capital cost and a decisive share of delivered performance. If under-specifying the network drops sustained GPU utilisation from, say, 85% to 55%, you have effectively discarded a third of the most expensive equipment in the building — permanently, on every job. For Indian buyers assembling on-premises clusters, two practical points follow: measure utilisation and collective completion time, not just link speed, when accepting a cluster; and treat cabling and optics as engineering rather than consumables, since a marginal transceiver produces exactly the intermittent degradation that is hardest to diagnose later. See our GPU cluster networking guide for the build-level detail.
Frequently asked questions
What is the difference between scale-up and scale-out?
Scale-up is the tightly coupled domain inside a node or rack — NVLink, NVSwitch or xGMI — where GPUs share memory at terabytes per second. Scale-out connects nodes over InfiniBand or Ethernet at hundreds of gigabits per second. Crossing that boundary is roughly an order of magnitude slower.
Why does packet loss matter so much in AI networks?
GPU communication uses RDMA, so a dropped packet forces a retransmission that stalls the collective operation in progress. Because every GPU waits for the collective to finish, one loss event idles the whole group — which is why AI fabrics are engineered for lossless behaviour.
What is a rail-optimised topology?
Each GPU index is assigned to a numbered rail, and same-index GPUs across all nodes connect to the same leaf switch. Since collectives mostly exchange data between same-index GPUs, traffic stays within a rail rather than traversing the spine, reducing hops and contention.
How much bandwidth does each GPU need?
Specify scale-out bandwidth per GPU rather than per node, and aim for non-blocking 1:1 leaf-to-spine where budget allows. Oversubscription shows up as congestion precisely during all-reduce, when every GPU is transmitting at once.
How do I know if my fabric is limiting the cluster?
Measure sustained GPU utilisation and collective completion times rather than raw link throughput. Utilisation well below the 80-90% range on a well-tuned job usually points at the fabric, cabling or congestion configuration rather than the GPUs.
Ready to deploy?
Talk to an RDP architect about power, cooling and lead time.