NCCL Tests and Collective Benchmarks: Validating an AI Fabric Before Production
Overview
nccl-tests is the standard open-source suite (from NVIDIA’s nccl-tests repository) for validating the communication fabric of a GPU cluster before production. It runs the collective operations training actually uses – all-reduce, all-gather, reduce-scatter – and reports bus bandwidth (busbw), a normalised figure you can compare directly against hardware limits. Commissioning practice is simple to state: run all_reduce_perf on one node, then 2, 4, 8 … up to full scale; on a healthy non-blocking fabric the large-message bus bandwidth stays near-flat as you scale, and any sag or outlier localises a bad link, optic or mis-wired rail.
Key takeaways
- Use bus bandwidth (busbw), not algorithm bandwidth, to compare against hardware specs – busbw normalises for the collective’s data-movement pattern, as documented in the nccl-tests performance notes.
- Single-node numbers validate NVLink/NVSwitch; multi-node numbers validate NICs, switches, optics and topology – always test both, in that order.
- As a reference point, published single-node H100 (NVLink 4) all-reduce results reach roughly 450-480 GB/s busbw at large message sizes; your platform’s quoted figure, minus 5-10 percent, is your acceptance floor.
- Scale in powers of two and plot busbw versus node count: a flat line is a healthy fabric, a cliff at a specific count localises the fault to the links or switch tier that entered at that step.
- Most “NCCL is slow” cases are one bad component: a flapping transceiver, a downtrained PCIe or IB link, a wrong rail mapping, or GPUDirect RDMA silently disabled.
Algorithm bandwidth vs bus bandwidth
all_reduce_perf prints two figures. Algorithm bandwidth (algbw) is message size divided by time – what the job perceives. Bus bandwidth (busbw) rescales algbw by the collective’s traffic factor (for ring all-reduce, 2 x (n-1)/n) so the number is comparable to link speed regardless of GPU count. That is why acceptance thresholds are always written in busbw: it answers “are the wires delivering what they are rated for?” A fabric can show falling algbw with rising GPU count while busbw stays constant – that is normal, not a fault.
What good looks like
Anchor expectations to your platform, not to folklore. Useful published reference points: single-node HGX H100 all-reduce has been reported around 450-480 GB/s busbw at large message sizes (see the discussion in nccl-tests issue threads); across nodes on a healthy 8-rail 400G fabric, large-message busbw should approach a large fraction of aggregate NIC line rate per node and, critically, hold near-flat from 2 nodes to full scale. Cloud operators publish validation procedures with the same shape – for example Crusoe’s InfiniBand validation guide. For acceptance, get the integrator to commit a busbw figure for your exact GPU count and message size, and test against it with an agreed 5-10 percent tolerance, as part of the layered plan in acceptance testing an AI GPU cluster.
The scaling ladder method
Run the same command at 1 node, then 2, 4, 8 and so on to full scale, sweeping message sizes (for example 8 B to 8 GB with -b/-e/-f) and recording large-message busbw at each rung. Three patterns matter:
- Flat curve: healthy fabric and correct rail wiring. Sign it.
- Cliff at a specific rung: the fault sits in whatever entered at that step – a specific leaf switch, a spine tier, or the inter-rack links. Bisect within that rung by testing node subsets.
- Uniformly low from 2 nodes: a systemic setting, not a cable – typically GPUDirect RDMA disabled (check NCCL_NET_GDR_LEVEL and nvidia-peermem), wrong NCCL topology detection, PCIe ACS enabled, or congestion control/PFC misconfigured on Ethernet fabrics.
Small-message latency deserves one pass too: it exposes switch and adaptive-routing problems that large messages hide, and it is what gates small collectives during inference and MoE routing. Rail wiring itself is covered in our guide to rail-optimised topology.
Diagnosing a bad link: a playbook
When one rung underperforms:
- 1. Find the slow pair. Run 2-node all_reduce_perf (or sendrecv/alltoall tests) across node pairs; the pairs containing the bad node or link will stand out.
- 2. Check physical counters. On the suspect ports, read symbol errors, link downed counts and effective link speed/width (ibstat or equivalent for InfiniBand; ethtool -S and switch telemetry for Ethernet). A link trained below rated speed or width is a defect even if it passes traffic.
- 3. Swap to isolate. Move the transceiver or cable to a known-good port: if the fault follows the optic, replace it; if it stays with the port, suspect the NIC or switch. Dirty fibre ferrules are a leading cause – clean and re-test before replacing hardware; see cabling and optics for AI fabrics.
- 4. Verify the software path. Confirm NCCL is using the interfaces you think (NCCL_DEBUG=INFO logs list chosen transports and rings), that GPUDirect RDMA is active, and that node-level PCIe links are at full generation and width.
- 5. Re-run the full rung and log the fix in the commissioning record – a link you repaired during acceptance is a link to watch during the first 90 days.
Reference numbers and thresholds
| Test scope | What it validates | Healthy signature (large messages) | Common failure if low |
|---|---|---|---|
| Single node, 8 GPU | NVLink / NVSwitch | Busbw near platform NVLink spec (H100-class: ~450-480 GB/s reported) | Bad NVLink lane, wrong module seating, ACS on |
| 2 nodes | NICs, first switch hop, GDR path | Busbw near single-node figure on rail-optimised 8-NIC nodes | GDR disabled, one NIC down/downtrained |
| 4-16 nodes | Leaf tier, rail wiring | Within ~5% of 2-node busbw | Rail mis-wire, one bad optic, oversubscription |
| Full scale | Spine tier, routing, congestion | Within 5-10% of quoted full-scale busbw | Adaptive routing/PFC/ECN misconfig, hot links |
| Small messages (<64 KB) | Latency path, switch config | Latency flat across pairs; no outlier pairs | Routing detours, buffer/QoS misconfiguration |
Frequently asked questions
Which nccl-tests binaries should acceptance actually run?
At minimum all_reduce_perf (dominant in data-parallel training), all_gather_perf and reduce_scatter_perf (dominant in FSDP/ZeRO sharding), and alltoall_perf if you will run MoE models. Sweep message sizes from a few KB to several GB; sign off on the large-message busbw and sanity-check small-message latency.
My algbw drops as I add nodes – is the fabric broken?
Not necessarily. Algorithm bandwidth falls with scale by construction for all-reduce, while bus bandwidth should hold roughly constant. Judge the fabric on busbw. If busbw itself sags with scale on a non-blocking fabric, then investigate.
Do I need MPI to run nccl-tests across nodes?
The usual method is mpirun (OpenMPI) launching one process per GPU across hosts, or running inside Slurm with srun. Containerised clusters commonly run it as a Kubernetes job. Whatever the launcher, keep it identical between the integrator’s demonstration and your witnessed re-run so numbers are comparable.
What NCCL settings matter most for a fair test?
Let NCCL auto-detect first and record NCCL_DEBUG=INFO output. The settings that most often explain bad numbers are GPUDirect RDMA state (nvidia-peermem loaded, NCCL_NET_GDR_LEVEL), interface selection (NCCL_SOCKET_IFNAME/NCCL_IB_HCA), and on Ethernet fabrics the RoCE congestion configuration. Document every variable used in the acceptance record.
How often should I re-run collective benchmarks after go-live?
Re-run a short ladder after any cabling change, firmware update or node swap, and on a monthly or quarterly cadence as drift detection – optics degrade and links retrain. Keep results with your baseline so a slow training job can be compared against known-good fabric numbers; day-2 practice is covered in our Day-2 operations article.
Ready to deploy?
Talk to an RDP architect about power, cooling and lead time.