Burn-In and Validation for GPU Clusters: Catching Failures Before Production
Overview
Burn-in is a deliberate multi-day stress campaign run on a new GPU cluster before production, designed to trigger infant-mortality failures – the weak GPUs, marginal optics, cold solder joints and leak-prone fittings that fail early on the bathtub curve – while the integrator is still on the hook. A practical burn-in runs sustained compute (HPL or gpu-burn class loads), memory stress, and fabric traffic (nccl-tests) continuously for 3-7 days at your site’s real ambient conditions, with every ECC event, XID error, link flap and thermal excursion logged and triaged. Nodes that misbehave get fixed or swapped before they can kill a production training run.
Key takeaways
- Burn-in exists because GPU failures follow a bathtub curve: a disproportionate share of hardware faults appear in the first days to weeks of sustained load, not spread evenly over the warranty.
- Run mixed stress, not one tool: HPL for compute and power delivery, DCGM diagnostics level 4 for memory, nccl-tests for fabric, and fio against local NVMe – each surfaces different defects.
- 3-7 days of continuous load is the working norm; anything under 24 hours is a smoke test, not a burn-in, because thermal soak failures need hours at temperature to develop.
- Watch counters, not just crashes: rising correctable ECC, row-remap events, PCIe replay counts and fabric symbol errors predict failures before anything visibly breaks.
- Define exit criteria in writing: zero uncorrectable errors, zero link flaps in the final 48 hours, node-to-node performance spread under about 5 percent, and a recorded baseline per GPU.
Why burn-in works: the bathtub curve
Electronics reliability follows the bathtub curve – elevated early-life (infant mortality) failure, a long flat middle, then wear-out. Manufacturing escapes such as marginal HBM stacks, imperfect cold-plate mounting, poorly seated connectors and borderline optics disproportionately fail under the first sustained thermal cycles. GPU cloud operators have made this discipline public: Crusoe reports burn-in testing every node before customer hand-off precisely to catch memory errors, misconfigured fabric ports and NVMe latency spikes that a quick functional check misses. When you buy a cluster outright, that burden lands on your commissioning plan – and it should run at your site, on your power quality and your ambient temperature, which in Indian server rooms in May is not the ambient it was tested at in a factory.
What to run: a mixed stress recipe
No single tool exercises everything. A sound recipe cycles or overlays:
- Compute and power: HPL or an equivalent dense-linear-algebra load pulls near-TDP power on all GPUs simultaneously – the hardest test of PSUs, busbars and facility power. Log per-GPU sustained clocks; a GPU clocking lower than its peers under identical load is throttling or defective.
- GPU memory: DCGM diagnostics with
dcgmi diag -r 4, which includes a memtest-style pattern test per NVIDIA’s DCGM documentation; watch for new row-remap entries (XID 63/64) and any uncorrectable ECC (XID 48). - Fabric: looped nccl-tests (all_reduce, all_gather, reduce_scatter) across the full cluster at large and small message sizes, which heats optics and stresses every link – see our companion piece on NCCL and collective benchmarks.
- Storage: fio write/read loops against local NVMe and the shared filesystem, plus a checkpoint-shaped burst test – covered in storage and data-path validation.
- A real training job: finish with 24-48 hours of an actual model run with checkpointing, which combines all of the above with the scheduler and drivers.
How long to run, and why
The point of duration is thermal soak and cycling. Solder and connector faults, marginal fan bearings, and coolant fitting weeps typically need hours at temperature and several heat-up/cool-down cycles to manifest. Working practice across operators clusters around 72 hours minimum, 5-7 days for large or liquid-cooled deployments, with at least two deliberate power cycles mid-campaign (cold restarts find a different fault class than steady state). The final 48 hours should be error-free; if a fix is applied on day 4, the clock restarts for the affected node. For liquid-cooled racks, burn-in doubles as the leak-detection soak – run it after the fill and pressure test described in commissioning a liquid-cooled rack.
Reading the logs: infant mortality signatures
Most early failures announce themselves in counters before they crash a job. Triage these during burn-in:
- XID errors in dmesg or DCGM: XID 48 (uncorrectable ECC), 63/64 (row remap recorded / failed), 79 (GPU fallen off the bus – often power or PCIe seating), 94/95 (contained/uncontained memory error). Any of these on a new GPU is grounds for replacement, not observation.
- Rising correctable ECC: a steady climb on one GPU while peers stay flat predicts trouble; NVIDIA’s row-remapping mechanism will spend spare rows, and pending remaps require a reset window.
- Fabric counters: symbol errors and link-down events on any port should be effectively zero after the first day; a port that flaps under thermal load is a bad transceiver, dirty ferrule or damaged DAC.
- Performance outliers: a node 5-10 percent slower than the fleet median at identical settings usually has a throttling GPU, a downtrained PCIe link, or a cooling defect. Straggler research on large clusters (for example the Guard paper) shows single slow nodes gate whole-job throughput, so catch them now.
Exit criteria and the day-1 baseline
Write exit criteria before you start: (1) zero uncorrectable errors and zero new row-remaps in the final 48 hours; (2) zero fabric link flaps in the final 48 hours; (3) node HPL spread within about 5 percent of median; (4) no thermal throttling at design ambient; (5) storage targets met under concurrent GPU load. On exit, snapshot the baseline – per-GPU serials, firmware, ECC and remap counters, sustained clocks and temperatures, per-port fabric counters – and archive it with the acceptance record from the acceptance test. That baseline is what day-2 monitoring drifts are measured against.
Burn-in schedule at a glance
| Phase | Duration | Load | Watch for |
|---|---|---|---|
| Smoke test | 2-4 h | DCGM diag r3, short HPL, single-node NCCL | DOA parts, miscabling, PCIe width |
| Thermal soak | 24-48 h | HPL loop near TDP, all nodes concurrently | Throttling, PSU trips, coolant weeps, ECC |
| Fabric soak | 24 h | Looped nccl-tests full-cluster + fio | Link flaps, symbol errors, slow nodes |
| Power cycling | 2 cycles | Cold shutdown and restart mid-campaign | Boot failures, drives/GPUs missing on return |
| Real workload | 24-48 h | Actual training job with checkpoints | NCCL timeouts, step-time variance, storage stalls |
| Clean window | Final 48 h | Continue mixed load | Must be error-free to exit |
Frequently asked questions
Is 24 hours of burn-in enough?
For a single workstation, possibly. For a server or cluster, no – thermal soak failures and connector faults often need multiple days and several heat cycles to appear. 72 hours is a defensible minimum for a production cluster; large or liquid-cooled deployments commonly run 5-7 days.
Does burn-in shorten the life of my GPUs?
Not meaningfully. Days at sustained load are within design operating conditions and are a tiny fraction of a GPU’s service life. The alternative – discovering a weak GPU three weeks into a month-long training run – costs far more in lost goodput than burn-in costs in hours.
What failure rate should I expect during burn-in?
Expect to find something on any deployment of scale: a few DIMM/HBM events, one or two marginal optics per few hundred links, occasionally a fan or PSU. Finding faults is the point; a burn-in that finds nothing on a large cluster more often reflects weak logging than perfect hardware.
Should I burn in at full cluster scale or node by node?
Both, in sequence. Node-level stress localises faults cheaply; full-scale concurrent load is the only way to stress facility power, cooling and the complete fabric at once. Run node-level first so full-scale failures point at shared infrastructure rather than individual parts.
Who runs burn-in – the integrator or my team?
Jointly, at your site. The integrator supplies and executes the tooling and fixes what fails; your engineer witnesses, holds the logs, and owns the exit criteria. If the integrator’s contract ends at delivery, negotiate a commissioning period that keeps them engaged through burn-in exit.
Ready to deploy?
Talk to an RDP architect about power, cooling and lead time.