Erasure Coding vs Replication for AI Datasets: Choosing per Data Class

Overview

Every byte in an AI storage tier is protected either by replication (full copies, typically 2x or 3x) or erasure coding (data split into k fragments plus m parity fragments, written as k+m across failure domains). The trade is capacity against write cost and rebuild behaviour: 3-way replication gives you 33 percent usable capacity with the cheapest writes and fastest rebuilds; an 8+2 erasure code gives you 80 percent usable capacity at the cost of read-modify-write penalties on small writes and heavier rebuild traffic. AI clusters should mix schemes per data class: replicate hot, small-write-heavy data; erasure-code the large sequential bulk that dominates capacity.

How the two schemes work

Replication writes complete copies to independent failure domains; any surviving copy serves reads and restores redundancy by a straight copy. Erasure coding (Reed-Solomon in nearly every storage product) splits an object into k data fragments, computes m parity fragments, and places the k+m stripe across distinct drives, nodes, or racks; any k surviving fragments reconstruct the data. Parallel filesystems and object stores expose this as pool or bucket policy – Lustre and BeeGFS traditionally lean on RAID within storage targets plus optional file-level mirroring, while WEKA, DAOS, Ceph, MinIO, and Storage Scale ECE implement distributed erasure coding natively. A long-running body of work, summarised well in the ACM survey on erasure coding for storage systems, underpins the durability claims vendors quote.

The capacity and durability arithmetic

Overhead is (k+m)/k for EC versus the replica count for replication. For 1 PB usable: 3x replication needs 3 PB raw; 8+2 EC needs 1.25 PB raw. At enterprise QLC pricing under 2026 allocation – see the memory and NAND squeeze – the difference funds a significant part of the GPU budget. Durability: 8+2 survives any two simultaneous fragment losses, same as 3-replication survives two copy losses, but EC spreads stripes across more devices, so at equal overhead (say 1.5x) EC schemes tolerate strictly more failure combinations. What actually determines annual durability in practice is rebuild speed versus failure rate – the window during which a second or third failure is fatal – which is why rebuild behaviour deserves more scrutiny than the nines on the datasheet.

Rebuild behaviour: where designs earn their keep

Losing a replica means copying full objects from surviving replicas – simple, sequential, fast. Losing an EC fragment means reading k fragments per stripe and recomputing – roughly k times the read traffic per rebuilt byte. Two design features rescue EC at scale: declustered placement (every drive holds fragments of many stripes, so hundreds of drives contribute a little each and a 30 TB drive rebuilds in hours, not days) and rebuild prioritisation of stripes with the least surviving redundancy. When evaluating a platform, ask for measured rebuild time on a full 30 or 61 TB drive under production load, and what read and write latency looks like during that rebuild – vendor answers vary far more than their durability nines do.

Write amplification: the flash angle

Full-stripe sequential writes cost EC almost nothing extra – the parity computation is cheap and the write is k+m fragments for k of data. Partial-stripe updates are the trap: modifying one block in a 4+2 stripe forces read-old-data, read-parity, recompute, rewrite – a multiple of the application write, landing on NAND whose endurance you already budgeted carefully (see all-flash NVMe design). Measurements on SSD arrays have shown random-write amplification tens of times worse under online EC than replication. AI saves you here: datasets, checkpoints, and archives are written as large sequential streams, EC’s best case. The residual small-write load – filesystem metadata, databases, KV caches, log files – is exactly what should sit on replicated or mirrored flash instead.

Recommendation by data class

Operational notes

Three things bite teams in production. First, failure-domain definition: 8+2 across 10 drives in one chassis protects against drive loss, not node loss – insist stripes span nodes (and racks where counts allow). Second, EC width versus cluster size: 16+4 needs at least 20 independent domains; small clusters must use narrower codes or accept correlated placement. Third, protection is not backup: erasure coding survives hardware failure, not an errant delete of a dataset bucket or a corrupted checkpoint propagating through replicas – keep an independent copy of irreplaceable data, a discipline that also aligns with DPDP-era expectations for data custody in Indian enterprises. For how protected tiers assemble into a full hierarchy, see tiering architecture for AI data.

Frequently asked questions