Network and Access Security for GPU Clusters: Segmentation, Jump Hosts and Secrets

Overview

An AI cluster is a high-value target for straightforward reasons: it concentrates the organisation’s most sensitive datasets, hardware worth crores that attackers can monetise directly through cryptomining, model weights that embody competitive advantage, and a dense web of credentials – registry tokens, storage keys, API secrets – that pipelines pass around by necessity. Publicly reported incidents against exposed ML infrastructure have included cryptojacking of GPU fleets and theft of credentials from job schedulers left open to networks they should never have faced. The defensible reference architecture is not exotic: four network zones, an RDMA fabric treated as a physically bounded trust domain, a single audited access path with MFA, and secrets that expire. This article sets out that baseline for an on-premises or colocated GPU cluster in India.

Why attackers want your GPU cluster

Three distinct attacker economics converge on AI infrastructure. Data theft: training corpora and RAG stores aggregate personal and proprietary data at exactly the density an exfiltration operation wants, with DPDP-scale consequences discussed throughout this category. Compute theft: a GPU node is directly monetisable, and security researchers have repeatedly reported cryptomining campaigns against exposed ML tooling – job schedulers, notebook servers and orchestration APIs with weak or absent authentication. Model theft: fine-tuned weights and the prompts and evaluation data around them are portable trade secrets. A cluster designed only against the first threat still loses money to the second and advantage to the third.

The four-zone segmentation model

A workable zone design for a GPU cluster:

• Management zone. BMCs, IPMI/Redfish, PDUs, switch management, firmware services. Reachable only from a hardened admin segment; no route from production or tenant networks; every session logged. Hardening detail in Supply Chain and Firmware Integrity for GPU Servers.
• Compute and fabric zone. GPU nodes and the east-west fabric. Node-to-node traffic is trusted within the zone; the zone boundary is where control lives.
• Storage zone. Parallel filesystems and object stores, exposed to compute through defined mounts and endpoints, with access logging – the audit layer described in Audit Evidence for AI Systems.
• Ingress zone. The only zone users and applications touch: jump hosts, API gateways for inference, dataset transfer landing areas. Everything crossing into compute or storage is authenticated, authorised and logged here.

Default-deny between zones, with the short explicit allow-list a GPU stack actually needs – scheduler, storage, telemetry – is achievable because cluster traffic patterns are unusually regular.

The RDMA fabric is a trust domain, not a LAN

InfiniBand and RoCE fabrics move tensors at hundreds of gigabits with minimal CPU involvement – and generally without encryption in transit, because line-rate crypto at fabric speeds has historically been impractical. The honest architectural response is to treat the fabric as a physically bounded trust domain: it should never extend beyond infrastructure you control; InfiniBand partition keys (or VLAN/VRF separation for RoCE) should separate tenants sharing a fabric; and anything plugged into the fabric – including DPUs and storage systems – is inside the trust boundary and must meet its bar. Where tenants must share nodes as well as fabric, the isolation mechanisms in Tenant Isolation on Shared GPUs complete the picture. If data must be protected even from the fabric operator, that is the confidential-computing use case rather than a network control.

One access path, strongly authenticated

Human access converges on a simple pattern: a bastion or jump host in the ingress zone, fronted by the corporate identity provider with MFA, issuing short-lived certificates rather than long-lived SSH keys, with session recording for privileged work. No shared accounts – individual attribution is both a security control and the evidence DPDP-style reviews request. Equally important is what never faces a general network: Slurm and Kubernetes API endpoints, MLflow and experiment trackers, Grafana and DCGM dashboards, notebook servers. Each has appeared in public reports of exposed-service compromises; all belong behind the ingress boundary. Service-to-service access uses workload identities with scoped tokens, not copied admin credentials.

Secrets: the soft underbelly of ML pipelines

AI pipelines are secret-hungry: object-storage keys, container-registry credentials, model-hub tokens, database passwords for RAG sources, inference API keys. The failure pattern is predictable – tokens hard-coded in notebooks, baked into container images, or exported in shell profiles, then leaked through an image push or repository. The baseline: a central secrets vault as the single source; short-lived, scoped credentials injected at job runtime by the scheduler or orchestrator rather than stored in code; automated scanning of repositories and images for embedded tokens; and revocation rehearsed, because a leaked token you can kill in minutes is an incident, while one you cannot inventory is a breach. Under BFSI-grade review, secrets handling is examined explicitly, as covered in BFSI Private AI GPU Server Controls.

Zones, contents and controls

Frequently asked questions