Tiering Architecture for AI Data: Hot NVMe, Warm Capacity, Cold Object

Overview

Most AI estates hold petabytes but train on terabytes: raw corpora, checkpoint history, and finished experiments dwarf the data any GPU touches this week. Tiering matches media cost to access heat – a hot TLC NVMe tier sized to the active working set plus checkpoint bursts, a warm QLC or hybrid capacity tier for recent datasets and checkpoint history, and a cold S3-compatible object tier for everything else. Done well, the expensive tier is a small fraction of total capacity and GPUs never notice; done badly, you either buy flash for archives or stall training on object-store latency. The design work is sizing the ratios and automating movement between tiers.

The three tiers and what lives in each

Hot (TLC NVMe, parallel FS or all-flash NAS): datasets for currently scheduled jobs, checkpoint landing and the most recent generation or two, hot embeddings and vector indexes serving production RAG (sizing in vector index sizing). Warm (QLC flash or dense hybrid, EC-protected): datasets used in the last quarter, checkpoint history retained for compliance or resumption, preprocessed intermediates too expensive to recompute instantly. Cold (object store): raw corpora, completed experiment artifacts, old checkpoint chains, and anything held for governance – which for Indian enterprises increasingly means DPDP-driven retention and locality requirements that on-prem object storage satisfies cleanly. Node-local scratch sits above all three as a disposable cache, covered in local NVMe scratch vs shared storage.

Deriving your ratios

Work from the training calendar, not the storage report. Sum the datasets attached to jobs planned for the next two sprints – that plus checkpoint working space is the hot tier floor. A worked example for a 64-GPU estate: 3 active datasets totalling 60 TB packed, checkpoint working space 2 generations x 4 TB, headroom 30 percent – a ~90 TB hot tier. If the organisation holds 2 PB total, hot is under 5 percent. Warm typically runs 3-6x hot; cold takes the rest. Two ratios to monitor quarterly: hot-tier occupancy (sustained above ~80 percent means demotion is failing or the working set grew) and promotion traffic (rising cold-to-hot reads mean the warm tier is missing the actual working set). Broader capacity planning context is in AI factory storage planning.

Movement mechanisms

Three mechanisms cover production needs. Policy demotion: filesystem HSM (Lustre HSM, Storage Scale ILM policies, WEKA tiering) or object lifecycle rules move data down by age and last-access automatically. Job-driven promotion: the scheduler stages a job’s dataset from warm or cold to hot (or node-local) before GPUs start – promotion should be an explicit, schedulable act, not a cache miss discovered at step one. Checkpoint drain: land on hot, replicate a generation, asynchronously copy history to warm/cold and prune. The emerging alternative is flattening: S3-over-RDMA platforms argue fast object storage can be the warm and hot capacity tier at once – the state of that argument is covered in object storage enters the training loop. It shrinks the stack but does not remove the need for an economic cold tier.

Failure patterns to design against

• The full hot tier: without enforced demotion and quotas, hot fills with dead data and new jobs stall on capacity, not bandwidth. Automate demotion from day one.
• Thrashing: aggressive age-based demotion plus a monthly-retrain workload equals constant re-promotion. Base policies on access patterns per dataset class, and let job metadata pin datasets scheduled for reuse.
• The silent cold read: a training job pointed at an unpromoted dataset runs – at object-store-over-WAN speed. Make tier residency visible in the scheduler and fail loudly instead.
• Restore-path amnesia: checkpoint history lands cold, then a resume needs generation N-3 and the restore takes hours. Test cold-restore time and include it in RTO maths.
• Metadata forgotten: tiering the data but not planning metadata capacity leaves listing and lookup slow regardless of media – the concern detailed in the small-file and metadata problem.

Tier design summary

Buying it without over-buying flash

The tiered design converts the NAND allocation problem into a smaller, better-specified one: instead of quoting petabytes of TLC, you buy a precisely sized hot tier (drive guidance in all-flash NVMe design), QLC for warm where its economics shine, and spindles or cloud for cold. Contract the growth path, not just day one: hot tiers grow with the training calendar, and mid-allocation expansions arrive on vendor timelines – the procurement dynamics in the 2026 memory and NAND squeeze apply directly. A hot tier at 60 percent occupancy with clean demotion is a healthier position than double the flash at 95 percent.

Frequently asked questions