GPU Health on Day 1: DCGM Diagnostics, XID Errors and the Baseline to Record

Overview

Day 1 of a new GPU cluster is the only time you will ever see the hardware in a known-clean state – so record it. A day-1 health baseline means every GPU has passed DCGM diagnostics at level 3 or 4, its ECC and row-remapping counters are captured, its sustained clocks and temperatures under load are logged, and its serials and firmware versions are inventoried. From then on, health is measured as drift from this baseline: a rising correctable-ECC count, a new XID in the logs, or a GPU clocking below its day-1 self are all early warnings that beat waiting for a training job to crash.

DCGM diagnostics: what each level tests

NVIDIA’s Data Center GPU Manager (DCGM) ships a layered diagnostic (dcgmi diag) documented in the DCGM user guide. Level 1 (-r 1) is a quick software and configuration sanity check (seconds). Level 2 adds PCIe and memory bandwidth tests (minutes). Level 3 adds sustained stress: targeted power, thermal and memory tests that catch marginal hardware (tens of minutes per GPU). Level 4, available since DCGM 2.4, adds a memtest-style pattern test over GPU memory – the closest thing to memtest86 for HBM. Commissioning practice: level 3 on every GPU as an acceptance gate, level 4 once during burn-in, and level 1 wired into the scheduler as a prologue/epilogue check. Archive the full JSON output per GPU, not a screenshot of “PASS”.

XID errors: the shortlist that matters

XIDs are error events the driver writes to the kernel log (dmesg) and DCGM surfaces; NVIDIA publishes the full XID catalogue. For commissioning and early life, the shortlist: XID 48 – double-bit (uncorrectable) ECC error; treat as hardware, expect a remap cycle or replacement. XID 63 – row remap recorded successfully; schedule the reset that activates it and watch for recurrence. XID 64 – row remap failed to record; the GPU cannot self-heal that address – open an RMA. XID 79 – GPU fell off the bus; on a new system this is usually PCIe seating, riser or power delivery, not the GPU die. XID 94/95 – contained/uncontained memory error on A100/H100-class parts; 95 means the job and possibly the node must restart. By contrast, XID 13 and 31 are typically illegal memory access by the application – a CUDA bug, not a hardware fault – and should not trigger a hardware swap. Community operations guides such as the ml-engineering debug notes track how operators triage these in practice.

ECC and row remapping, briefly

HBM on datacenter GPUs is ECC-protected: single-bit errors are corrected silently (counted as correctable), double-bit errors are detected and kill the affected computation (uncorrectable). Since A100, GPUs handle recurring faults by row remapping – swapping a weak memory row for one of a finite pool of spares, persisted in the InfoROM (see NVIDIA’s memory error management documentation). What this means operationally: correctable ECC counts are a trend signal, not an alarm – but a GPU whose correctable count climbs while its peers stay flat is announcing a future failure. Baseline aggregate counters on day 1 (nvidia-smi -q -d ECC,ROW_REMAPPER or DCGM fields), require zero pending remaps at acceptance, and write a policy now: how many remap events before you demand a swap under warranty, so the conversation with your vendor is about an agreed threshold rather than feelings.

Thermals, clocks and the performance baseline

A GPU can pass every functional test and still be a straggler. Under identical sustained load (HPL or a fixed benchmark), record for each GPU: steady-state SM clock, HBM temperature, hot-spot/junction temperature where exposed, board power, and any clock-throttle reasons flagged (nvidia-smi and DCGM expose throttle masks – thermal, power, and HW slowdown). On a healthy, well-cooled node all GPUs settle within a narrow band; one GPU clocking 5-10 percent lower is a cooling defect (bad paste, blocked airflow, cold-plate flow restriction) or a marginal part – exactly what burn-in exists to catch. This performance baseline also arms your monitoring: dcgm-exporter feeds the same fields to Prometheus, so day-2 dashboards can literally alert on “slower than day 1”, the operating practice covered in Day-2 operations for rack-scale AI.

The day-1 record: exactly what to archive

Per GPU: serial number and PCI address, VBIOS and InfoROM versions, driver and CUDA versions, DCGM diag level 3/4 JSON results, aggregate ECC counters, row-remap state (zero pending, remap history), sustained clock/temperature/power under the reference load, and NVLink status/error counters. Per node: BMC and NIC firmware, PCIe link width/generation per device, PSU health, and fabric port counters. Store it in version control or your CMDB next to the acceptance certificate – it is the reference every warranty claim and every “is this node slow?” question will be answered against, and it belongs in the integrator’s handover pack alongside the as-built documentation.

Day-1 health checks and thresholds

Frequently asked questions