The Small-File and Metadata Problem in AI Training Storage

Overview

A storage system that streams 100 GB/s can still starve GPUs when the dataset is 50 million 100 KB images. The reason is that every file open is a metadata operation – lookup, permission check, attribute fetch – and metadata operations are served by a different, usually much smaller, part of the storage system than bulk reads. For small-file workloads the numbers that matter are metadata ops per second and small-block IOPS, not GB/s. The fixes are known: pack small files into large shards, scale or distribute the metadata service, and stage hot data onto node-local NVMe.

Where the time actually goes

Opening a file on a POSIX parallel filesystem involves a path lookup, permission and attribute checks, and often a layout query telling the client which data servers hold the content – each a network round trip to a metadata service. For a 10 GB file, that one-time cost is noise. For a 100 KB file it can exceed the transfer time itself: at 1 ms of effective metadata latency, a dataloader thread manages barely 1,000 files per second regardless of how fast the data path is. Multiply by shuffled access order (defeating readahead) and thousands of dataloader workers across a cluster, and epoch time becomes a metadata benchmark. Storage vendors’ own workload analyses – for example the ResNet-50 pattern used in MLPerf Storage, which reads ~100 KB samples – exist precisely because this pattern dominates vision and multimodal training.

How metadata services scale

Architectures split into two camps. Centralised: Lustre’s MDS/MDT design serves metadata from dedicated servers; scaling means adding MDTs via DNE (Distributed Namespace) and placing them on flash, which works but requires deliberate namespace design (striped directories, careful placement of hot directories). BeeGFS behaves similarly with its metadata services. Distributed: WEKA, DAOS, and IBM Storage Scale spread metadata handling across many or all nodes, so metadata throughput grows with the cluster. The practical consequence: on a centralised design, one hot directory with 10 million files lands on one server; on a distributed design it is sharded. Our parallel filesystem comparison covers which camp each product sits in.

IOPS and metadata ops versus GB/s

Sequential bandwidth is the cheapest number for a vendor to make large, which is why it headlines datasheets. For small-file training the load profile is different: high 4-128 KB random read IOPS on the data path, and tens of thousands of opens, stats, and closes per second on the metadata path. When evaluating, ask for (or measure) three numbers: file creates per second, stats per second, and 64 KB random read IOPS at your target client count. A system quoting 200 GB/s but 50,000 metadata ops per second will train ResNet-class workloads worse than a smaller system quoting 20 GB/s and 500,000 metadata ops. Measurement method is covered in benchmarking AI storage.

Fix it at the source: pack the dataset

The highest-leverage fix costs no hardware. Packing formats bundle thousands of samples into large files read sequentially: WebDataset (tar shards, streaming-friendly, PyTorch-native), TFRecord (TensorFlow lineage), Parquet or Arrow for tabular and text, FFCV and similar loaders for vision. A 50-million-image dataset becomes perhaps 5,000 shards of 1 GB; file opens drop by four orders of magnitude and the storage system sees the sequential streams it was built for. The costs are real but modest: a preprocessing pipeline, loss of per-file random access (mitigated by shard-level shuffling), and re-packing when the dataset changes. Almost every large training operation converges on packing; doing it before you size storage can shrink the metadata requirement dramatically.

Fix it in the platform: cache and stage

Where packing is impractical – long-tail research datasets, many small teams, code and Python environments – platform mitigations help: metadata on dedicated NVMe (mandatory in any 2026 design), client-side caching of attributes and directory entries, and staging hot datasets onto node-local NVMe before the job starts, which converts shared-metadata load into local ext4/xfs operations that are an order of magnitude cheaper. The staging pattern is detailed in local NVMe scratch vs shared storage.

What matters per workload

Measuring before you buy

Two tools give you the truth cheaply. mdtest (part of the IOR suite) measures file create, stat, and delete rates at scale and is the metadata component of the IO500 benchmark; run it at your real client count, not from one node. fio with 4-64 KB random-read profiles at realistic queue depths covers the data side. Then replay a slice of your actual dataloader – a one-hour PyTorch run with your dataset layout – because attribute caching, readdir behaviour, and client tuning interact in ways synthetic tools miss. Insist on running these during any proof of concept; a vendor unwilling to let you run mdtest is telling you something.

Frequently asked questions