NVMe over Fabrics for AI Storage: RoCE vs TCP vs Fibre Channel

Overview

NVMe over Fabrics (NVMe-oF) lets a GPU node access remote NVMe drives over a network with latency close enough to local flash that the difference stops mattering for most AI workloads. The transport decision is the real design choice: RDMA over Converged Ethernet (RoCEv2) adds only single-digit microseconds per I/O but demands a lossless-tuned network; NVMe/TCP runs on any Ethernet and any kernel but adds tens of microseconds and CPU load; NVMe over Fibre Channel is mature and deterministic but confines you to an FC estate few AI shops build today. Disaggregation pays when it lifts flash utilisation from the 50-60 percent typical of stranded local drives to the 85-95 percent a shared pool can reach.

What NVMe-oF actually changes

iSCSI carried SCSI – a protocol designed for disk – over TCP, with one queue and heavy per-I/O overhead. NVMe-oF extends the native NVMe command set, with its thousands of parallel queues, across a network fabric. The result is that a JBOF (just a bunch of flash) or storage server can present namespaces to many GPU nodes at near-local performance, and vendor-published measurements such as Western Digital’s RoCE-vs-TCP white paper show remote flash tracking local flash closely when the fabric is built properly. This is the plumbing underneath most modern all-flash AI storage platforms, whether or not the marketing mentions it.

Transport option 1: RDMA (RoCEv2 and InfiniBand)

RDMA transports move data directly between NIC and memory without kernel copies. Published figures put InfiniBand’s added latency around 2-5 microseconds per I/O and RoCEv2 around 5-10 microseconds – effectively invisible next to the drive’s own access time. The cost is fabric discipline: RoCEv2 wants a lossless network with Priority Flow Control and ECN correctly configured on every switch, and misconfiguration produces pathological behaviour (pause storms, congestion spreading) that is hard to debug. If your team already runs a lossless fabric for GPU east-west traffic, extending that competence to storage is natural; if not, budget real engineering time.

Transport option 2: NVMe/TCP

NVMe/TCP runs over ordinary Ethernet with ordinary switches and has been in mainline Linux for years. Typical added latency is in the tens of microseconds – commonly quoted ranges run 30-80 microseconds over RDMA depending on congestion and CPU load – and it consumes host CPU cycles for the TCP stack. For dataset reads, checkpoint writes, and most AI I/O patterns, that is acceptable; the workloads that genuinely feel the difference are small-block, latency-critical patterns like database pages or very hot cache tiers. DPUs such as NVIDIA BlueField-3 can terminate NVMe/TCP or NVMe/RDMA in hardware, reclaiming host cores – a pattern covered in GPUDirect Storage and DPU offload.

Transport option 3: NVMe over Fibre Channel

NVMe/FC maps NVMe onto an existing FC SAN (32G/64G FC today). It inherits FC’s strengths – lossless by design, mature zoning and multipathing, deterministic behaviour – and its weaknesses: a separate fabric, separate skills, and per-port costs that rarely make sense for a greenfield AI build. The honest guidance: choose NVMe/FC when a bank or enterprise already operates FC at scale and wants AI storage inside that governance model; choose Ethernet transports otherwise.

Building the latency budget

Work end to end: drive media time plus controller, plus fabric hops, plus transport overhead, plus host stack. A useful planning skeleton for a 4KB random read: NVMe TLC media and controller about 60-90 microseconds; two switch hops about 1-4 microseconds; RDMA transport 5-10 microseconds or TCP 30-80 microseconds; host completion path 5-15 microseconds. The lesson most architects take from writing this down is that the drive dominates until you pick TCP under congestion – at which point the network becomes the story. Measure with fio at target queue depths rather than trusting any of these numbers; our benchmarking guide covers method.

Transport comparison

When disaggregation beats local drives

Keep flash local when the data is disposable scratch, when the cluster is small, or when you cannot fund fabric engineering – the trade-offs are laid out in local NVMe scratch vs shared storage. Disaggregate when drives sit half-empty across many nodes, when a node failure must not strand data, when checkpoint targets need more write bandwidth than one server’s slots can hold, or when you want to grow capacity without buying servers. Under the current NAND allocation environment – see the 2026 memory and NAND squeeze – pooling scarce drives efficiently is itself a procurement argument, and one that resonates for Indian buyers watching import lead times.

Frequently asked questions