Checkpoint Storage at Frontier Scale: Sizing for 2026-2028

Overview

Checkpointing is the least glamorous line in a training cluster design and one of the most expensive to get wrong. Published analyses put checkpoint overhead at 12 to 43 percent of total training time at scale, and a 16,000-accelerator cluster has been reported to take roughly 155 checkpoints per day. That is not a storage detail — it is a direct tax on the GPU hours you paid for. This article works through the arithmetic: how large a checkpoint is, how often you must write one, what bandwidth that implies, and how to retain them without unbounded capacity growth.

How big is a checkpoint, really

Start with parameters. Model weights in BF16 are 2 bytes per parameter. But a resumable checkpoint must also carry optimiser state, and Adam-family optimisers keep two moment estimates plus, commonly, an FP32 master copy of the weights. That lands the total at roughly 12 to 16 bytes per parameter for a conventional full checkpoint. A 70-billion-parameter model is therefore around 0.8 to 1.1 TB per checkpoint, and a 400-billion-parameter model is in the multi-terabyte range.

Two things shrink this. Sharded checkpointing writes each data-parallel rank’s slice rather than gathering the full state to one process, which parallelises the write across nodes. And optimiser state sharding, already standard in ZeRO-style and FSDP training, means the state is already distributed — the checkpoint simply persists it in place. Both are now default in mature training stacks and should be assumed in any sizing.

Frequency is a reliability question, not a storage one

How often you checkpoint follows from how often the job dies. The reported figures are sobering: Llama 3 pre-training on 16,384 GPUs saw 466 job interruptions over 54 days, with mean time to failure around 2.7 hours, and ByteDance’s MegaScale reported recovering from failures over 100 times during a production run on more than 10,000 GPUs. At that failure rate, checkpointing every few hours means routinely discarding an hour or more of cluster-wide work per incident.

The classical result applies: optimal checkpoint interval scales with the square root of the product of mean time to failure and checkpoint cost. Practically, teams at scale checkpoint every few minutes to tens of minutes, which is only tolerable if the write itself is cheap. That is why the engineering effort has moved from “checkpoint less often” to “make checkpointing nearly free”.

The bandwidth arithmetic

These are order-of-magnitude figures to frame a conversation, not a substitute for measuring your own stack. The pattern they show is the important part: required write bandwidth scales linearly with model size, and the storage tier that comfortably serves dataset reads may be nowhere near sufficient for synchronous checkpoint writes. This is precisely the case where the accelerated data path described in GPUDirect Storage and DPU offload earns its cost.

Asynchronous and tiered checkpointing

The standard mitigation is to decouple the GPU pause from the storage write. The GPU copies state into host memory or a local NVMe staging tier quickly, training resumes, and a background process persists it to shared storage. Recent research has pushed further: FFTrainer, for example, uses surplus network capacity to save and load state in a way that reduces reported MFU loss to no more than 0.27 percent, effectively removing checkpointing from the critical path.

A practical tiered policy looks like this. Write a frequent, cheap checkpoint to node-local NVMe or host memory for fast recovery from the common single-node failure. Write a less frequent durable checkpoint to shared storage for recovery from correlated failures and for retention. Keep the fast tier small and the durable tier policy-driven. This mirrors the tiering logic used for inference caches in KV cache offloading.

Retention: the capacity side

Frequency drives bandwidth; retention drives capacity. Keeping every checkpoint from a multi-week run at multi-terabyte sizes produces petabytes of largely useless data. A defensible policy keeps the last few checkpoints for rollback, one checkpoint per epoch or per fixed token count for scientific reproducibility, and every checkpoint that was ever used to produce a released artefact. Everything else expires.

Retention also has a governance dimension. For regulated Indian workloads, a checkpoint contains the model state derived from training data, and where it is stored is part of the residency story. Keeping the durable checkpoint tier in-country, with a documented lifecycle, is far simpler to evidence than reconstructing what was retained after the fact — the same discipline covered in AI factory storage planning.

What to specify in a build

Four numbers belong in the specification. Required sustained write bandwidth to the checkpoint tier, derived from model size, interval and the overhead percentage you will accept. Capacity for the retention policy, with headroom for a doubling of model size within the cluster’s life. Endurance class for the drives, since frequent large writes consume flash endurance quickly and the current NAND pricing environment makes premature drive replacement expensive. And recovery time objective — how long a full restore takes, which is the read side and is often untested.

Add one acceptance test: a timed restore from a durable checkpoint into a running job. Teams routinely validate that checkpoints write and never validate that they read back into a working state at speed. The first time you discover a restore takes 40 minutes should not be during an incident. For the broader planning context, see GPU storage planning for LLM checkpoints and RAG indexes.

Frequently asked questions