GPUDirect Storage and DPU Offload: Designing the AI Data Path

Overview

The AI data path is the sequence a byte travels from flash to a tensor core, and in 2026 two technologies are reshaping it: GPUDirect Storage, which lets NVMe write straight into GPU memory without a host bounce buffer, and DPU offload, which moves the storage stack itself off the CPU. The short answer for architects is that these matter most when the pipeline is throughput-bound and CPU-starved — large sequential reads, checkpoint writes and KV cache paging — and matter very little when the bottleneck is somewhere else. This article covers what each does, what to specify, and how to tell whether you need it.

What the conventional path costs

Without acceleration, reading a training shard works like this: the NVMe device DMAs into a page-cache buffer in host memory, the CPU copies that into a pinned staging buffer, and a second DMA moves it to GPU memory. Every byte crosses the host memory bus at least twice and occupies a CPU core to orchestrate. On an eight-GPU node saturating tens of gigabytes per second, that becomes a meaningful fraction of the CPU complex — cores that were bought to run the data loader, augmentation and tokenisation, not to shuffle bytes.

The symptom in production is a GPU utilisation graph that plateaus below expectation while CPU sits pegged. Teams often misdiagnose this as a storage bandwidth problem and buy more drives, when the constraint is the copy path. This is the same class of misdiagnosis described in storage architecture for AI training.

What GPUDirect Storage eliminates

GPUDirect Storage (GDS) establishes a direct DMA path between the storage device or NIC and GPU memory, removing the host bounce buffer. The CPU still issues the I/O, but it no longer touches the data. Two effects follow: host memory bandwidth is freed for the work that actually needs it, and latency drops because a copy stage disappears from every transfer.

GDS is not automatic. It requires a supported filesystem or block client with a GDS-aware driver, a compatible framework path (cuFile or a library built on it), and drives or a fabric that can DMA to GPU memory. If any link in that chain falls back to the conventional path, you get the conventional performance — silently. Verifying that the accelerated path is actually in use, rather than assuming it, is the single most common gap we see in commissioning.

What a DPU adds on top

A DPU is a programmable device that runs the infrastructure stack on its own cores. NVIDIA’s BlueField-4 pairs a Vera CPU with a ConnectX-9 class 800 Gb/s interface and is positioned as a storage architecture, handling I/O scheduling, protocol conversion, data verification, compression and encryption with no host CPU involvement, and supporting NVMe-oF, GDS and the DOCA acceleration libraries.

Two deployment shapes matter. On the client side, running the parallel filesystem or object client on the DPU isolates the storage stack from tenant workloads — useful in multi-tenant clusters where a noisy neighbour currently steals cycles from everyone. On the target side, a DPU inside a flash enclosure turns it into a self-contained storage node with no separate storage server, which is the model behind the JBOF and AI-native storage products announced through 2025-26.

Which workloads actually benefit

The KV cache row deserves emphasis. As inference deployments offload attention caches to a fast tier, that path becomes latency-critical and sustained rather than bursty — a good fit for an accelerated data path. We cover the workload itself in KV cache offloading.

A specification checklist

When writing a tender or a build spec, name the whole path rather than a single component. At minimum: NVMe drives with the endurance class matched to checkpoint write volume; a fabric with RDMA (InfiniBand or RoCE) sized for aggregate node throughput, not per-node peak; NICs or DPUs with the GDS support level stated explicitly; a filesystem or object client with a named GDS-capable driver version; and a framework data loader path that uses cuFile rather than POSIX reads.

Add two acceptance tests to the contract. First, a measured throughput test with GDS enabled and disabled, so the accelerated path is proven rather than claimed. Second, a CPU utilisation measurement during that test — the whole point is that host cores stay free. Without those tests, a compliant-on-paper system can quietly run the slow path in production.

Sizing guidance for Indian deployments

For a single 8-GPU training node, node-local NVMe plus a conventional network filesystem is usually sufficient and GDS is a nice-to-have. The economics change at cluster scale: once shared storage feeds dozens of nodes concurrently, the copy overhead is multiplied across every node and the CPU tax becomes a real line item. That is the point at which an accelerated path pays for the added complexity.

One local caution. DPU-based storage products are early, with several reaching general availability through 2026, and support depth in India varies by vendor. Confirm local spares, firmware support and a named escalation path before committing a production tier to first-generation hardware. The capacity and throughput arithmetic in AI factory storage planning should drive the design first; the acceleration layer is an optimisation on top of a correctly sized system, never a substitute for one.

Frequently asked questions