AI: Security

Tenant isolation and segmentation in shared AI clusters

Miro Kosut

Miro Kosut

— 9 min read
Share

Requirement / business driver

Safely share extremely expensive GPU infrastructure across tenants/teams without lateral movement or data leakage between them.

Why it's hard

GPUs are too costly to dedicate per tenant, so clusters are shared - which means each tenant's compute, RDMA traffic, and storage must be isolated. And because the fabric can't be inspected inline (Securing the high-performance fabric), isolation rests on segmentation and identity, not a firewall in the middle.

Mechanisms

The validated converged-stack pattern makes the isolation grains per layer concrete. Adding an AI platform to a shared converged pod is executed as adding a tenant, and the tenant's whole footprint is: a VLAN set of its own (management plus each storage lane), a switch-side SVI and VRF for the tenant's routed traffic, a dedicated storage virtual machine (SVM) on the shared array - its own volumes, logical interfaces per VLAN, its own admin credential and audit domain - and a compute decision. Two forks are argued explicitly. Dedicated or shared hosts: the validation dedicates hosts to each tenant rather than mixing tenants on every hypervisor - simpler blast-radius and capacity accounting at the cost of pooling efficiency. Where tenants boot from: booting every host from the infrastructure tenant's SVM reuses one server profile and address pool; per-tenant boot SVMs buy stronger separation but cost a parallel set of profiles, initiator pools, and boot policies - the validation judged that overhead not worth it. The design lesson: tenancy on converged infrastructure is a bundle of per-layer grains (VLAN/VRF at the network, SVM at the array, host groups at compute) that must be added coherently, not a single switch knob.

The container and orchestration layer

Most shared AI clusters are Kubernetes-scheduled, which stacks a software tenancy boundary on top of the fabric and GPU mechanisms above. Namespaces, RBAC, and network policy isolate tenants logically, but a container is a weaker boundary than a VM - kernel-sharing means a container escape lands on a host that owns GPUs and RDMA NICs. Map the isolation need to the mechanism: MIG for hardware-isolated GPU slices, vGPU/time-slicing where consolidation beats isolation, dedicated node pools per tenant when kernel-sharing itself is unacceptable (AI-enabling hardware - GPU, DPU, SmartNIC). Add the supply chain: signed and scanned images, and secrets for model weights and data credentials held in a secrets manager, never baked into images (Model and data protection). The shared-responsibility split moves with hosting: on-prem you own every layer; on cloud/managed Kubernetes the provider owns the control plane while tenant isolation inside the cluster stays yours (AI security design and hosting models - air-gapped, private, public).

The complementary tenancy grain is the cluster itself. Instead of subdividing one shared cluster ever more finely, the validated fleet pattern gives workloads or teams their own clusters - bare-metal, virtualised, cloud-hosted, or edge - and recovers control through a central management plane (Rancher-class enterprise container management): every certified Kubernetes distribution registers into one point that unifies authentication, RBAC, policy, and upgrades, with policy-as-code guardrails (Kubewarden-class) pushed fleet-wide so misconfiguration cannot drift per cluster. The grain trade is real: cluster-per-tenant buys hard blast-radius and lifecycle independence at the cost of per-cluster overhead the management plane must absorb. And when the compliance bar is regulatory, the distribution choice itself becomes the lever - a hardened, DISA-STIG-certified engine (RKE2-class) makes the platform's baseline certifiable rather than hand-assembled.

Between namespace tenancy and cluster-per-tenant sits a middle grain the validated Rancher design names explicitly: one shared control plane fronting per-tenant worker-node sets, separated by labels/taints, each set with its own VLANs and SVM. It spends the fewest control-plane nodes, but the caveat is the design point - every worker set must satisfy the one master set's Kubernetes version, so upgrades and maintenance windows become cluster-wide events negotiated across all tenants; cluster-per-tenant is what buys independent lifecycle cadence back. Two network rules complete the validated tenant bundle: storage lanes (iSCSI/NFS/NVMe-TCP VLANs) stay unrouted - reachable only by L2 adjacency inside the tenant, invisible to any router - while only the tenant's external and VM networks are routed, behind ACLs/firewalls (or mapped onto transfer networks toward cloud/remote sites); and where tenants self-manage their own nodes, the shared in-band management network is replaced by per-tenant management networks, so the operator path itself stops being a shared surface.

The allocation grain: whole servers or GPU groups

Beneath every mechanism above sits one enterprise decision: is the tenancy grain the server or the GPU group? Server-level allocation hands a tenant whole nodes - all eight GPUs and the intact NVLink domain (plus PXN across rails where enabled) - the simplest isolation, and in a dedicated training and fine-tuning cluster usually the right call: any job, however small, runs fully accelerated and JCT stays shortest; the price is pooling efficiency. GPU-level allocation splits a dense node across tenants, in practice in multiples of two, and raises utilisation - but on an NVLink-meshed server, putting the GPU NICs into different VRFs is not isolation: the GPUs still reach each other peer-to-peer over NVLink inside the chassis. The bare-metal Kubernetes mechanism that actually severs that path is host-level: cgroup device visibility restricts the container to its assigned GPUs (cross-tenant NVLink peers simply do not exist to it) and the orchestrator injects CUDA_VISIBLE_DEVICES so the CUDA runtime sees only the same set - fabric segmentation for the wire, OS-level device scoping for the in-chassis mesh. Sub-GPU slicing (MIG) is rarely relevant here - training spans many GPUs across nodes, so the grains that matter are node and group, not fractions. The bar itself also moves with the audience: enterprise tenants are internal teams, so the posture can be deliberately softer than a GPUaaS platform serving external customers - shared resources (storage above all) reached over route-leaked shared VRFs are acceptable inside one organisation where a GPU cloud would forbid them; a hybrid estate (training plus inferencing) is where right-sizing the grain gets genuinely hard.

Recommendation / justification

Use DPU-enforced isolation plus per-tenant overlay/VRF/SGT segmentation and separate storage, governed by zero-trust microsegmentation. Justify by the shared-cluster economics and the no-inline-inspection constraint - isolation and identity are the primary tools.

What would change this (mid-scenario twist)

  • Strict tenant separation / regulated tenants -> stronger isolation (dedicated rails/pods or DPU-enforced per-tenant policy).
  • Lateral-movement risk -> zero-trust microsegmentation across the fabric.

Validation checks

  • Is each tenant's compute, RDMA traffic, and storage isolated (no lateral path)?
  • Is isolation enforced by segmentation/identity (DPU/overlay/SGT), given no inline inspection?

IPv6 / dual-stack note

Tenant segmentation (VRF/SGT/overlay) must cover v4 and v6 equally - an unsegmented v6 path defeats isolation.

The GPU-cloud isolation grains

When the shared cluster hardens into a GPU cloud - the NCP reference shape - the service stack layers as IaaS -> CaaS -> PaaS (bare-metal/VM provisioning, managed Kubernetes, a tenant-facing AI platform), and each layer fixes an isolation grain. Bare metal gives a tenant the strongest node isolation - no neighbours and the least operator access - but its price is more than money: provisioning effort, and a turnover sanitisation duty every time a node changes tenants (wipe memory, return all firmware to a known-good state, erase the local ephemeral drives). Hypervisor-hosted VMs fit the majority of needs at lower cost, provided the hypervisor is hardened against resource-contention, memory-leak and CPU cache-line attacks from hostile neighbours. Inside a VM, container (runc-class) process isolation is weaker than the hypervisor's, so the platform should let each tenant choose one or many workloads per worker VM. Two alignment rules bound the model: keep each node to a single tenant wherever possible - at once the most secure arrangement and the one that lets the node's GPUs cooperate unimpeded (sharing a node demands a hardened hypervisor or DPU-enforced isolation) - and a worker VM belongs to exactly one tenant and one control plane. The control plane is per-tenant too: a dedicated K8s control-plane instance (KaaS) whose API surface is never shared between tenants, with the pragmatic concession that the physical hosts carrying those control-plane VMs may be shared for utilisation. Networking-side tenant isolation lives in NVIDIA's gated Common Networking reference (sales-channel only) - plan for it, don't assume it.

Spaced repetition

Why is tenant isolation in shared GPU clusters primarily a segmentation/identity problem?

GPUs are too expensive to dedicate per tenant, and the fabric can't be inspected inline - so isolation rests on DPU enforcement + overlay/VRF/SGT segmentation, not a mid-fabric firewall.

The NIC-level tool for per-tenant isolation in AI clusters is [...].

The NIC-level tool for per-tenant isolation in AI clusters is the DPU (per-tenant policy, control plane separate from host).

Containing lateral movement across a shared AI fabric uses [...].

Containing lateral movement across a shared AI fabric uses zero-trust microsegmentation (SGT/overlay per tenant).

Why treat containers as a weaker tenancy boundary than VMs in a shared GPU cluster?

They share the host kernel, so an escape exposes the GPUs and RDMA NICs on that node; compensate with MIG (hardware GPU isolation), per-tenant node pools, K8s RBAC and network policy, signed/scanned images, and a secrets manager for model weights.

What is the validated per-layer footprint when adding an AI platform as a tenant to shared converged infrastructure?

A VLAN set of its own plus a switch-side SVI/VRF, a dedicated storage SVM (own volumes, LIFs, admin, audit), and a host decision - dedicated hosts per tenant and boot-from-the-infrastructure-SVM were the validated calls; tenancy is a bundle of per-layer grains added coherently.

When does cluster-per-tenant beat subdividing one shared AI cluster?

When blast-radius and lifecycle independence matter more than pooling efficiency - a central management plane (Rancher-class ECM) registers every certified distribution and unifies authentication, RBAC, policy-as-code guardrails, and upgrades fleet-wide; a STIG-certified engine can make the compliance baseline itself certifiable.

The NCP node-to-tenant rule?

Keep each node aligned to a single tenant - single-node multi-tenancy is inherently less secure and splits the GPUs' peer cooperation; if a node must be shared, a hardened hypervisor or DPU-enforced isolation is mandatory. A worker VM belongs to exactly one tenant and one K8s control plane.

Bare metal vs VM vs container as the GPU-cloud isolation grain?

Bare metal isolates best (no neighbours, least operator access) but pays in provisioning and turnover sanitisation - memory wipe, firmware to known-good state, ephemeral-drive erase; hypervisor-hardened VMs fit most needs at lower cost; in-VM (runc-class) containers isolate weakest, so let the tenant choose one-or-many workloads per worker VM.

Per-tenant KaaS: the API-bearing K8s control plane is [...], though the physical hosts running the control-plane VMs [...] for utilisation.

Per-tenant KaaS: the API-bearing K8s control plane is never shared between tenants, though the physical hosts running the control-plane VMs may be shared for utilisation.

In the tenancy grain ladder, what does a shared control plane with per-tenant worker sets trade away?

Lifecycle independence - every worker set must match the single master set's Kubernetes version, making upgrades cluster-wide events negotiated across tenants; cluster-per-tenant restores independent cadence at the cost of per-cluster overhead.

A validated tenant network bundle keeps [...] while only the external/VM networks are routed behind ACLs or firewalls.

A validated tenant network bundle keeps storage lanes (iSCSI/NFS/NVMe-TCP VLANs) unrouted - L2-adjacent only while only the external/VM networks are routed behind ACLs or firewalls.

Why is VRF separation of the E-W NICs insufficient for GPU-level multi-tenancy on a dense node, and what closes the gap?

NVLink-meshed GPUs reach each other peer-to-peer inside the chassis regardless of the network; host-level scoping closes it - cgroup device visibility limits the container to its assigned GPUs, and CUDA_VISIBLE_DEVICES limits the CUDA runtime to the same set.

Server-level vs GPU-level allocation in a dedicated training cluster - the default and why?

Server-level: every job gets whole nodes with the intact NVLink domain, so all jobs run fully accelerated and JCT stays shortest. GPU-level (in multiples of two) buys utilisation at the cost of host-level isolation work - and MIG is rarely relevant because training spans nodes, not GPU fractions.

Sources

  • Cisco AI POD for Enterprise Training and Fine-Tuning Design Guide (Jan 2026) - server- vs GPU-level allocation grain, cgroup + CUDA_VISIBLE_DEVICES NVLink severing, enterprise-vs-GPUaaS trust posture; full 94-page pass 2026-07-26 (user-supplied PDF).
  • NVIDIA NCP Software Reference Guide, Workload Isolation (docs.nvidia.com, Mar-2026, fetched in full) + Introduction: isolation grains, node-to-tenant rule, KaaS, sanitisation, IaaS/CaaS/PaaS ladder.
  • Cisco/NetApp/SUSE, FlexPod Datacenter with SUSE Rancher for AI Workloads design guide (Dec 2023) - fleet/ECM multi-cluster pattern, RKE2 STIG lever; shared-control-plane worker-set grain with version coupling, unrouted storage lanes, per-tenant management networks; full 75-page pass 2026-07-26 (user-supplied PDF).
  • Cisco/NetApp, FlexPod Datacenter with Generative AI Inferencing (CVD, Feb 2024) - multi-tenant tenant-addition pattern, per-layer grains, host and boot-SVM forks; design chapters passed 2026-07-25.
  • Cisco AI/ML blueprint; DPU multi-tenancy and TrustSec/EVPN segmentation guides.

domain: AI-Infra · blueprint-ref: AI-Infra 3.1 Hosting implications (multi-tenancy / isolation) · type: design-decision · status: complete · tags: [elective/ai-infra, ai/security, tradeoff/security, tradeoff/scale]

Read more