Parallel Filesystems Compared: Lustre, BeeGFS, WEKA, Storage Scale, DAOS
Overview
Five parallel filesystems dominate AI cluster shortlists in 2026: Lustre, BeeGFS, WEKA, IBM Storage Scale (the product formerly known as GPFS and Spectrum Scale), and DAOS. The short version: Lustre and BeeGFS give you open-source throughput with real administrative effort attached; WEKA and Storage Scale are commercial products that trade licence cost for small-file performance and supportability; DAOS posts the highest IO500 numbers but has the smallest operational ecosystem. The right choice depends less on peak GB/s than on your metadata profile, your team’s storage skills, and who answers the phone at 2 a.m.
Key takeaways
- Lustre remains the HPC throughput reference and is free to run, but it is widely regarded as the most administration-heavy option; budget for dedicated Lustre expertise or a supported distribution such as DDN EXAScaler.
- BeeGFS is the pragmatic open-core middle ground: simpler to deploy than Lustre, good striped throughput, but weaker small-file and metadata behaviour than flash-native designs.
- WEKA was built NVMe-first and is strong on mixed and small-file AI workloads; it requires its own client software and a commercial licence.
- IBM Storage Scale is the enterprise all-rounder with mature snapshots, tiering, and multiprotocol access, at enterprise licence pricing.
- DAOS leads published IO500 lists and suits object-native, Optane-successor architectures, but production deployments outside national labs remain rare.
Architecture at a glance
All five distribute data across many servers, but they differ in how they split data and metadata:
- Lustre separates Metadata Servers (MDS/MDT) from Object Storage Servers (OSS/OST) and stripes files across OSTs. Parallelism scales with OST count, which is why large sequential reads fly and metadata can become the choke point.
- BeeGFS uses a similar split (management, metadata, storage services) but its services are userspace daemons that can co-exist on fewer nodes, which is why small teams find it easier to stand up.
- WEKA runs a fully distributed, flash-native filesystem in userspace (DPDK/SPDK data path), with metadata sharded across all nodes rather than concentrated on dedicated servers.
- IBM Storage Scale is a symmetric clustered filesystem with distributed locking and token management, plus mature policy-driven tiering and multiprotocol (NFS/SMB/S3/CSI) front ends.
- DAOS abandons the POSIX-block heritage entirely: it is an object store built for NVMe and persistent memory, with POSIX presented as a layer on top.
Throughput versus metadata behaviour
For large sequential streams – checkpoint writes, shard reads of preprocessed datasets – all five can saturate modern networks when sized correctly; published IO500 bandwidth results for tuned Lustre, DAOS, and WEKA systems all reach hundreds of GB/s. The separation shows up on metadata-heavy and small-file work: millions of file opens per epoch, directory scans over image corpora, Python environments on shared storage. Flash-native, distributed-metadata designs (WEKA, DAOS) and Storage Scale with distributed metadata generally behave better here than classic Lustre or BeeGFS layouts, where a small number of metadata servers can bottleneck. If your training data is millions of small files rather than packed shards, weight metadata ops per second above GB/s – covered in depth in our article on the small-file and metadata problem.
Operational burden: the honest view
This is where shortlists are usually decided. Lustre expertise is a genuine specialisation: kernel-module clients, careful version matching, and recovery procedures that punish inexperience. Most production Lustre today runs as a vendor-supported appliance (DDN EXAScaler, HPE ClusterStor) precisely to outsource that burden. BeeGFS is notably friendlier – many university clusters run it with generalist Linux admins – but you still own tuning and failure handling unless you buy ThinkParQ support. WEKA and Storage Scale shift the burden to a vendor relationship: you pay licence and support, and in exchange get tested upgrade paths and someone accountable. DAOS assumes a sophisticated operations team; treat it as a research investment unless you have one.
Client software and GPU data path
Lustre and Storage Scale use kernel clients; BeeGFS ships a kernel module with a relatively easy build; WEKA requires its userspace client on every GPU node. All of the mainstream options now support RDMA transports and NVIDIA GPUDirect Storage paths to move data NIC-to-GPU without bouncing through system memory – see GPUDirect Storage and DPU offload for how that changes the design. Check client-kernel compatibility against your OS image discipline before committing: a filesystem whose client blocks your CUDA driver upgrade cadence becomes an organisational problem, not a storage one.
Comparison summary
| Filesystem | Design | Metadata model | Best fit | Operational burden | Cost model |
|---|---|---|---|---|---|
| Lustre | MDS/OSS striping, kernel client | Dedicated MDS (DNE for scale-out) | Large sequential HPC/AI throughput | High; specialist skills or vendor appliance | Open source; support via DDN/HPE |
| BeeGFS | Userspace services, striping | Dedicated metadata services | Mid-size clusters, lean teams | Moderate | Open core; paid ThinkParQ support |
| WEKA | Flash-native distributed FS | Fully distributed | Mixed/small-file AI, cloud bursting | Low-moderate; vendor managed | Commercial licence |
| IBM Storage Scale | Symmetric clustered FS | Distributed with token mgmt | Enterprise AI plus multiprotocol | Moderate; mature tooling | Commercial licence |
| DAOS | NVMe-native object store | Distributed key-value | Leadership-class performance | High; small ecosystem | Open source |
Choosing for your cluster
For a 4-16 node GPU cluster, a supported BeeGFS or an all-flash NFS platform is usually enough, and the filesystem decision matters less than drive and network sizing – see AI factory storage planning. From roughly 32 nodes up, checkpoint burst bandwidth starts to dictate the choice: work backwards from checkpoint size and write-window targets as described in checkpoint storage at frontier scale. Indian buyers should also weigh support reach: Lustre and Storage Scale skills exist in the HPC labs ecosystem, while commercial options depend on the vendor’s local partner bench. Whatever you shortlist, insist on a proof-of-concept run with your own workload traces rather than vendor-published benchmarks.
Frequently asked questions
Is Lustre still relevant for AI, or is it an HPC legacy?
Still relevant. Most of the world’s largest AI training clusters run Lustre derivatives because nothing beats its cost per GB/s at extreme scale. The caveat is operational: plan for a supported distribution or dedicated expertise.
Do I need a parallel filesystem at all for a small GPU cluster?
Often not. Below roughly 8 nodes, local NVMe scratch plus a solid NFS server or object store frequently feeds GPUs adequately. Parallel filesystems earn their complexity when many nodes must read and checkpoint concurrently.
Which option handles millions of small files best?
Distributed-metadata, flash-native designs – WEKA, DAOS, and well-configured Storage Scale – generally publish stronger small-file and metadata results than classic Lustre or BeeGFS layouts, though Lustre DNE and metadata-on-flash narrow the gap.
What does WEKA cost compared to open-source options?
WEKA is licensed commercially (typically per-capacity subscription); exact pricing is quote-based. Open-source Lustre or BeeGFS have no licence fee but shift cost into engineering time and optional support contracts. Model both over three years before comparing.
Can these filesystems tier to object storage?
Yes – Storage Scale, WEKA, and Lustre (via HSM tools) can tier cold data to S3-compatible object stores, which pairs naturally with the hot/warm/cold architecture described in our tiering article.
Ready to deploy?
Talk to an RDP architect about power, cooling and lead time.