Acceptance Testing an AI GPU Cluster: What to Verify Before Sign-Off
Overview
An acceptance test is the gate between “the integrator says it works” and “you now own it.” For an AI GPU cluster, acceptance means every node passes hardware diagnostics, every fabric link carries its rated bandwidth, agreed benchmark numbers (HPL, nccl-tests) are met, and a real multi-hour training job completes without faults – all recorded against written pass/fail thresholds before anyone signs. Clusters that skip this step routinely discover miscabled rails, downgraded PCIe links and marginal GPUs weeks later, when the vendor’s crew has left and the burden of proof has shifted to you.
Key takeaways
- Never accept a cluster on “it powers on and trains” – insist on a written acceptance test plan (ATP) with numeric pass/fail thresholds agreed before delivery.
- Test in four layers: per-component (DCGM level 3/4 diagnostics), per-node (HPL, local NVLink bandwidth), fabric (nccl-tests at increasing scale), and full-cluster (a real training job of 12-48 hours).
- Demand per-GPU evidence: serial numbers, firmware versions, ECC and row-remap counters at zero or documented, and DCGM pass logs archived as part of the handover pack.
- Performance thresholds should be relative to the vendor’s own published or quoted numbers – typically within 5-10 percent for HPL and nccl-tests bus bandwidth at full scale.
- Sign-off is layered: integrator signs the test results, your engineer countersigns witness of execution, and the budget holder signs final acceptance that releases payment and starts the warranty clock.
Why a written acceptance test plan matters
An acceptance test plan (ATP) is a short document, agreed with the integrator before hardware ships, that lists every test, the tool and command used, the numeric threshold, and the evidence to be archived. Its value is contractual as much as technical: once payment terms reference the ATP, “the cluster is slow” becomes a measurable dispute rather than an argument. Deployment specialists such as Crusoe have published how nodes that fail pre-delivery testing are triaged and re-tested rather than shipped – your ATP applies the same discipline at your site, on your power and your cooling, which is where marginal hardware actually shows itself.
For Indian buyers there is a second reason: warranty and AMC clocks usually start at delivery or installation. A dated, signed acceptance certificate fixes that start point and documents the as-delivered health baseline, which matters if you later claim an early-life failure. See our companion guide on the first 90 days and early-life failures.
The four layers of acceptance testing
Structure the ATP bottom-up so a failure is localised before you scale the test:
- Layer 1 – Component: every GPU passes DCGM diagnostics at level 3 (level 4 adds a memtest pass, per NVIDIA DCGM documentation). ECC error and row-remap counters recorded; any non-zero value documented and accepted explicitly or the GPU swapped.
- Layer 2 – Node: single-node HPL (or vendor burn-in suite) sustained for hours at expected TFLOPS; nvbandwidth or single-node nccl-tests confirming NVLink/NVSwitch bandwidth; PCIe links trained at full width and generation (a Gen5 x16 device sitting at x8 is a classic assembly defect); all fans, PSUs and BMC sensors clean.
- Layer 3 – Fabric: nccl-tests (all_reduce_perf and friends) at 2 nodes, then powers of two up to full scale. Bus bandwidth should stay near-flat as you scale on a healthy non-blocking fabric; a sagging curve points to a bad link, optic or rail mis-wire. Details in our guide to NCCL and collective benchmarks.
- Layer 4 – Cluster: a representative training job (for example a NeMo or Megatron run, or your own model) for 12-48 hours with checkpointing enabled, no XID errors, no NCCL timeouts, and step-time variance within a few percent.
Performance acceptance: HPL, nccl-tests and a real job
Synthetic benchmarks catch different faults than real jobs, so you need both. HPL stresses compute and power delivery and gives a single comparable number per node – reject nodes more than about 5 percent below the fleet median. nccl-tests stress the fabric in the communication patterns training actually uses. The real training job is the only test that exercises storage, scheduler, drivers and fabric together; research on large training runs (for example the OLMo 2 report) consistently shows failures that only appear under sustained mixed load. Do not sign off on screenshots: require raw logs, and re-run at least one test per layer yourself with your own engineer driving the keyboard.
Health acceptance: DCGM, ECC and firmware
Performance can be on-target while health is marginal. Before sign-off, capture per GPU: serial number, VBIOS, driver and CUDA versions, InfoROM state, aggregate ECC counters, row-remap history (pending remaps are a red flag), and thermal behaviour under sustained load – no thermal throttling at your room’s design ambient temperature. Capture per node: BMC/firmware versions, PSU health, NIC firmware, and link error counters on every fabric port after the burn-in window (they should be effectively zero after cabling settles). This snapshot becomes your day-1 baseline; our article on GPU health and diagnostics covers what to record and why.
Sign-off criteria and who signs what
Layered sign-off keeps accountability clear. The integrator’s engineer signs each test record as executed. Your infrastructure engineer countersigns as witness – never accept unwitnessed results for the fabric and cluster layers. Facilities signs power and cooling readiness (see facility readiness for 100 kW racks). Finally the budget holder signs the acceptance certificate, which should state the ATP version, all waivers (any threshold not met but accepted), the warranty start date, and the punch list of open items with dates. Withhold an agreed percentage of payment – 10-20 percent is common – until the punch list closes.
Acceptance test matrix
| Layer | Tool / method | Pass criterion (typical) | Who signs |
|---|---|---|---|
| Component (GPU) | dcgmi diag -r 3 (or -r 4) | All tests pass; ECC and row-remap counters zero or waived in writing | Integrator + witness |
| Node compute | HPL / vendor stress suite, 2-4 h | Within 5% of fleet median TFLOPS; no throttling, no XID | Integrator + witness |
| Node interconnect | nvbandwidth, single-node nccl-tests | NVLink bandwidth at platform spec; PCIe at full width/gen | Integrator + witness |
| Fabric | all_reduce_perf 2 nodes to full scale | Bus BW within 5-10% of quoted; near-flat scaling curve; zero link errors | Integrator + network engineer |
| Cluster | Real training job, 12-48 h | Completes with checkpoints; no NCCL timeouts; stable step time | Your ML lead |
| Final acceptance | Certificate + punch list | All layers passed or waived; docs handed over | Budget holder |
Frequently asked questions
How long should acceptance testing take?
For a single 8-GPU node, one to two days including a sustained stress run. For a multi-rack pod, plan one to two weeks: component and node layers run in parallel across nodes, but fabric scaling tests and the long training job are serial. Do not compress the sustained-load window – most marginal hardware only fails after hours at temperature.
What benchmark numbers should I put in the contract?
Reference the vendor’s own quoted or published figures for your exact configuration, with a tolerance (typically 5-10 percent) rather than absolute numbers from marketing material. For nccl-tests, specify message size, GPU count and the expected bus bandwidth at full scale, since small-message latency and large-message bandwidth stress different things.
Can I do acceptance testing without stopping at synthetic benchmarks?
You should. Synthetic tests localise faults; only a real training or inference workload validates the whole path including storage and scheduler. Industry practice is to finish with a representative job of at least 12 hours with checkpointing, and compare achieved step time against the vendor’s sizing claims.
What if the cluster fails one test?
Log it, localise it (which node, which link), and let the integrator fix and re-run that layer plus the layers above it. Track failures on a punch list attached to the acceptance certificate. Accept with a written waiver only if the shortfall is understood, bounded, and priced – for example a service credit or extended warranty.
Does acceptance replace burn-in?
No – they are complementary. Burn-in is a longer soak (days) designed to trigger infant-mortality failures; acceptance is a structured pass/fail gate for contract and payment. Many teams run them together: burn-in runs 3-7 days, and the acceptance certificate is signed on the burn-in evidence plus the layered tests. See our guide to burn-in and validation.
Ready to deploy?
Talk to an RDP architect about power, cooling and lead time.