Day-2 Operations for Rack-Scale AI: Failures, Monitoring and Service

Overview

Buying a rack-scale AI system is a project; operating it is a discipline. The best public data point on what awaits: Meta reported 419 unexpected interruptions in 54 days while pretraining Llama 3 405B on up to 16,384 H100s — roughly one interruption every three hours at that scale, dominated by GPU and HBM faults. A single NVL72-class rack is far smaller, but it concentrates 72 GPUs, 18 NVSwitch chips, thousands of liquid-cooling joints and a dense cable plant into one enclosure, and researchers note component counts make failure rates per system higher than 8-GPU nodes, not lower. Day-2 readiness — monitoring, spares, checkpoint discipline and service workflow — decides whether the rack delivers its benchmark economics or bleeds them into downtime.

What actually fails, and how often

Fleet studies and operator playbooks agree on the ranking: GPU and HBM memory errors first (including silent-degradation modes like row-remapping exhaustion and rising correctable-error rates), then interconnect — flapping NVLink lanes, optics and cables — then software/driver faults, with liquid-cooling incidents rare but high-consequence. Two operational truths follow. First, failures cluster early: burn-in at commissioning (48–72 hours of full-load stress with DCGM diagnostics) surfaces infant mortality before production does. Second, degradation precedes failure: a GPU whose ECC correctable errors trend upward or whose clocks throttle under stable thermals is announcing its exit — fleets that act on these signals convert crashes into scheduled swaps.

The monitoring stack that earns its keep

The base layer is NVIDIA DCGM: continuous background health checks, on-demand deep diagnostics (memory stress, compute stress, PCIe and NVLink bandwidth verification), and Prometheus-exportable metrics. Layer on: thermal and power telemetry per tray correlated with facility CDU data (a rising GPU-temperature-at-constant-load curve is a coolant-flow problem, not a GPU problem); NVLink/fabric error counters with per-lane granularity; and job-level correlation so a failed training step maps to a component in minutes. Operator experience collected in guides like Introl’s monitoring playbook credits systematic monitoring with up to ~70% reduction in failure-related downtime — the highest-ROI line in the ops budget.

Checkpoint discipline: the arithmetic of interruptions

For training and fine-tuning, decide checkpoint cadence by expected-loss math: if an interruption costs the time since the last checkpoint plus restart overhead, and your system’s interruption rate is once per N hours, checkpointing every 30–60 minutes bounds the loss to trivial while a daily checkpoint gambles a shift’s compute. The enabling requirement is storage bandwidth — writing a multi-terabyte state in minutes rather than an hour — which is a first-class sizing input in our checkpoint storage guide. For serving fleets, the equivalent discipline is graceful drain: health-triggered removal of a degrading GPU from the serving pool before it fails mid-request.

Servicing a liquid-cooled monolith

Modern rack-scale designs made the right serviceability trade: compute trays connect to the cooling manifold through quick-disconnect fittings, so a tray swap does not drain the loop, and deployment guides such as Leviathan’s GB300 commissioning walkthrough describe tray-level replacement as the standard field operation. What that still requires: technicians trained on wet-mate procedures and torque specs, leak-detection verification after every intervention, ESD and lifting equipment for 30–60 kg trays, and — the India-specific line — spares stocked in-country. A tray that must clear customs turns a four-hour MTTR into a two-week outage; make spare-pool location and replenishment SLA explicit contract terms, alongside the facility obligations covered in our facility readiness guide.

Operational readiness checklist

Frequently asked questions