Training Goodput at 10,000 GPUs: Failures, MFU and Honest Throughput

Overview

The metric that decides whether a large training cluster earns its capital is goodput — useful work delivered against compute requested — and at scale it diverges sharply from peak specifications. The published record is clear about why: Llama 3 pre-training on 16,384 GPUs recorded 466 job interruptions across 54 days with mean time to failure around 2.7 hours, and ByteDance’s MegaScale reported recovering from more than 100 failures during a production run on over 10,000 GPUs while achieving 55.2 percent model FLOPS utilisation on a 175-billion-parameter transformer across 12,288 GPUs. This article explains what erodes goodput and what to do about it.

Three metrics that get conflated

Peak FLOPS is a datasheet number: what the silicon can do under ideal conditions. Model FLOPS utilisation (MFU) is the share of that peak which actually advances training, accounting for memory bandwidth limits, communication, bubbles in pipeline parallelism and kernel inefficiency. Goodput takes one further step and subtracts time lost to failures, restarts, rollback to the last checkpoint, and degraded operation.

The gap between them is not small. A cluster quoted at some exaflop figure, running at 50 percent MFU with 90 percent availability, delivers well under half its headline number in useful work. Budgeting a training programme against peak FLOPS is the single most common planning error we see, and it typically produces a schedule that slips by a factor of two.

Why failures scale with cluster size

A synchronous training job is a single distributed process: if one participant dies, the job dies. With thousands of GPUs, plus their host systems, NICs, optical transceivers, cables, power supplies and cooling, the aggregate failure rate is the sum of thousands of component rates. Even excellent per-component reliability produces frequent cluster-level events once you multiply it out.

The failure mix is instructive. GPU faults including memory errors are the largest single category in published analyses, followed by network and optical link failures, then host and software issues. Notably, a significant share are transient — a link flap or a correctable error storm — which argues for automated detection and restart rather than human triage. The operational discipline is similar to what we describe for inference fleets in day-2 operations for rack-scale AI.

Stragglers: the failure that does not alert

A crashed GPU is obvious. A slow one is not. Because data-parallel training synchronises at every step through an all-reduce, the whole job proceeds at the pace of its slowest participant. A GPU thermally throttling, a NIC negotiating a degraded link speed, or a node with a misconfigured NUMA binding will silently cap cluster-wide throughput while every health check reports green.

This is why per-rank step-time telemetry is not optional. The detection signal is variance: collect per-step duration per rank, alert on outliers rather than on absolute thresholds, and correlate with GPU clock, temperature, ECC counters and link speed. Teams that add this instrumentation routinely find they have been losing double-digit percentages of throughput to a handful of degraded nodes.

What erodes goodput, and what to do

The last row is where hardware generation helps directly. A wider NVLink domain lets more of the model sit inside one rack, reducing reliance on pipeline parallelism — one reason the 144-GPU domain in Vera Rubin NVL144 matters more for goodput than its FLOPS figure suggests.

Recovery time is the dominant lever

Given a fixed failure rate you cannot control, goodput is determined by how much work each failure costs. That has two components: work lost since the last checkpoint, and time to get running again. Frequent asynchronous checkpointing addresses the first — see checkpoint storage sizing for the arithmetic. The second is often larger and less examined: job re-initialisation, weight reload across thousands of ranks, and the ramp back to steady state can take tens of minutes if nobody has optimised it.

Recent research targets exactly this. FFTrainer, for instance, uses surplus network capacity for state management and reports MFU loss no greater than 0.27 percent from failover, effectively making recovery nearly free. Whether or not you adopt a specific system, the design principle transfers: keep recovery state close and warm, maintain a pool of hot spare nodes so a replacement does not wait on provisioning, and rehearse recovery so it is a routine automated path rather than an incident.

What to put in an acceptance test

Specify goodput, not peak. A credible acceptance test runs a representative model for a sustained period — days, not minutes — and reports achieved MFU, total wall-clock, cumulative interruption count, mean time to recovery, and per-rank step-time variance. It should include a deliberate fault injection: kill a node mid-run and measure the time back to steady-state throughput.

Two further clauses are worth including. A straggler test that verifies per-rank telemetry exists and that a degraded node is detected automatically. And a network validation across the full fabric under collective load, since link-level problems often appear only at scale — the topology considerations are covered in GPU cluster networking. A cluster that passes a ten-minute benchmark and fails a five-day run is a common and expensive outcome.

Frequently asked questions