Benchmarking AI Storage: IO500, fio, mdtest, and MLPerf Storage

Overview

Storage benchmarks answer different questions, and buying decisions go wrong when a number from one is used to answer another. fio measures raw device and filesystem I/O; IOR and mdtest (combined in the IO500) measure parallel-filesystem bandwidth and metadata at scale; MLPerf Storage measures the only thing that ultimately matters – whether the storage keeps simulated accelerators busy on real training I/O patterns. An honest evaluation uses all three layers, run by you, on the configuration you would actually buy, at the client count you would actually run.

Layer 1: fio – devices and single clients

fio is the flexible workhorse: block size, queue depth, read/write mix, direct I/O, and parallelism are all scriptable. Use it to (a) verify drives deliver datasheet behaviour in your chassis (a thermal-throttling Gen5 drive shows up here first), (b) characterise single-client throughput to shared storage – one GPU node’s view – and (c) probe latency at fixed queue depths. Three profiles cover most AI questions: 1 MB sequential read at QD32+ for dataset streaming, 4-64 KB random read for small-sample workloads, and 1 MB sequential write sustained for at least your full checkpoint duration. Report p99 latency alongside bandwidth; averages hide the tails that stall dataloaders.

Layer 2: IOR and mdtest – the parallel scale test

IOR drives coordinated multi-node bandwidth (MPI-launched, matching how training jobs actually hit the filesystem); mdtest hammers metadata – creates, stats, deletes – across many clients. The IO500 combines both into a ranked list, which is useful for two things: comparing filesystem architectures at the high end, and stealing its methodology. Run IOR in both easy (large sequential, friendly alignment) and hard (small unaligned I/O) modes; the gap between them is a measure of how much your workload’s shape matters on that platform. The same easy/hard split in mdtest exposes whether metadata performance survives realistic directory structures. Which architectures tend to win which sub-tests is discussed in our parallel filesystem comparison.

Layer 3: MLPerf Storage – workload realism

MLPerf Storage emulates accelerators running real training workloads (3D U-Net’s large sequential reads, ResNet-50’s many ~100 KB samples, and since v2.0, checkpoint write/restore patterns) and reports how many simulated accelerators a storage system can keep above a utilisation threshold. Because submissions are peer-reviewed under MLCommons rules, it is the hardest benchmark for a vendor to game – and the published result nearest to your purchase decision. Caveats remain: submissions use vendor-chosen hardware scale, so normalise per storage node or per rack unit, and check the accelerator model emulated (an H100’s demand differs from a B200’s). If a vendor on your shortlist has no MLPerf Storage submission, ask them to run it in your PoC.

How vendor numbers mislead

• Cache-warm reads: quoted read bandwidth served from DRAM or a hot flash cache, not the media. Fix: read a dataset larger than total cache, cold.
• Compressible data: platforms with inline compression quote bandwidth on zeros. Fix: fio with random (incompressible) buffers.
• Peak-not-sustained writes: SLC caches and deferred garbage collection make 10-minute writes look great. Fix: write for your full checkpoint window repeatedly.
• Hero client counts: aggregate GB/s achieved with hundreds of tuned clients tells you nothing about your 16 nodes. Fix: demand per-client and at-your-scale numbers.
• Metadata silence: no opens/stats per second anywhere in the datasheet. Fix: mdtest in the PoC; see the small-file and metadata problem for why this ceiling matters.

Benchmark-to-question map

Running an honest bake-off

A defensible PoC procedure: (1) Fix the configuration – the exact node counts, drive SKUs, and network you would buy; no vendor lab specials. (2) Run the three layers in order – fio locally and per-client, IOR/mdtest at full client count, then either MLPerf Storage or a one-hour replay of your own training job with your dataset layout. (3) Test degraded states: pull a drive mid-run, reboot a storage node, and measure performance during rebuild – production lives there more often than datasheets admit. (4) Measure the checkpoint cycle end to end: write, drain, and a full restore, since restore bandwidth gates recovery time (sizing context in checkpoint storage at frontier scale). (5) Record everything – tool versions, mount options, firmware – so the winning numbers are reproducible on the delivered system, and make acceptance contractual against that record. For Indian procurements running vendor bake-offs remotely, insist on shell access to the test rig rather than screenshared dashboards; the mount options tell you more than the graphs.

Frequently asked questions