AI: Network

Connectivity models and SDN in AI fabrics

Miro Kosut

Miro Kosut

— 10 min read
Share

Requirement / business driver

Decide how the AI fabric is structured and operated - a dedicated island vs part of the DC, traditional routing vs SDN, single vs multi-site.

Connectivity models (2.8)

  • Isolated / purpose-built island - a dedicated back-end fabric for GPU/RDMA traffic, separate from front-end, management, and storage planes. This is the norm for the high-performance GPU network.
  • Integrated in the main DC - CLOS, two- or three-tier hierarchical, sharing the broader DC fabric.
  • Single-site (typical) vs multi-site (DCI for distributed training - costly and latency-bound, Bandwidth and capacity planning).

When is a converged fabric acceptable? The discriminator is per-host GPU/NIC density. Standard rack servers with 1-2 GPUs (inference, RAG, fine-tuning farms) present a couple of 100G ports each; GPU, front-end, and storage traffic can share one fabric with per-class QoS (a lossless class for RoCE) - a dedicated island buys little. Dense 8-GPU training nodes present 8x400G of pure collective traffic per host and force the dedicated back-end: no shared fabric absorbs that cleanly, and the split also gives an independent failure/change domain - the front-end stays the flexible general-purpose network (BGP or EVPN) while the back-end stays plain-routed and lossless.

Infrastructure and operation (2.4-2.6)

  • Traditional - a BGP-routed CLOS (commonly eBGP underlay) - simple, scalable, deterministic.
  • SDN / controller-based (e.g., Cisco ACI) - policy/automation and integration/interconnection with other fabrics.
  • Layer 2 (2.5) - kept minimal; the back-end is L3-routed to avoid L2 scale/loops, with reachability propagated by the IGP/BGP; EVPN (Overlays and BGP EVPN) where an overlay is needed.
  • Routing optimisations (2.6) - tuned eBGP for CLOS, fast convergence (BFD/TI-LFA), ECMP, and entropy for the fabric (BGP in enterprise design).

Pod networking is a design choice, not a given. The default CNI in the validated OpenShift builds is OVN-Kubernetes: an OVN controller and Open vSwitch on every node build a Geneve overlay between nodes, so pod-to-pod traffic crosses the physical fabric as opaque encapsulated flows - simple, but the fabric team sees tunnels, not tenants. The alternative the same designs name is the ACI CNI, which extends the fabric's SDN policy model into the pods - one policy domain from leaf to container - trading CNI neutrality for fabric-native visibility and enforcement. The fork decides who owns east-west container policy: the platform team (overlay) or the fabric team (SDN-integrated CNI).

The attachment domain has its own sizing arithmetic. A fabric-interconnect-class domain scales to the order of 160 servers, but the real bound is the port budget: every port spent attaching servers is a port not available as an uplink, so growing the domain raises its oversubscription toward the network - the same fan-in trade the SAN edge prices explicitly (Storage networking - FC, FCoE, iSCSI, and NVMe-oF). And domains are facility fault domains: splitting compute across multiple domains per room or hall bounds the blast radius of environmental events - a reason to run several smaller domains even when one big one still fits the port math.

Operating models: self-managed, SDN-controller, and as-a-service

The traditional-vs-SDN split above is really a three-point operating-model spectrum, and which point you pick changes who runs the fabric and how far it reaches. Self-managed NX-OS is box-by-box - each switch configured (often from templates) and operated by the customer's own team: maximum control, maximum operational burden. On-prem SDN controller (Cisco ACI, or NDFC-driven fabrics) centralises policy and automation on a controller the customer hosts - less per-box toil and central intent, but the controller is one more system to run and integrate. Cloud-managed fabric-as-a-service (e.g. Cisco Nexus HyperFabric) moves the controller into the vendor's cloud and delivers the fabric turnkey: the customer designs and operates through a cloud portal while Cisco runs the controller, and that one control point spans primary DC, colocation, and edge sites. For AI this model usually ships as an integrated, AI-native stack - validated switches plus accelerators, SuperNICs, and a data platform pre-fitted together (AI-enabling hardware - GPU, DPU, SmartNIC, Storage and checkpointing for AI) - trading bespoke control for fastest time-to-operational. The design consequence is a control/dependency trade: as-a-service minimises in-house operational expertise and speeds deployment, but the management plane now depends on vendor-cloud reachability and the provider's lifecycle, and lock-in shifts from the box to the operating model. Crucially the data plane stays local - a cloud-resident controller schedules and configures, it does not carry GPU collective traffic - so a controller-reachability outage hits change and visibility, not the running RDMA fabric.

The spectrum has a further pole and a hard constraint worth naming. Left of self-managed NX-OS sits the customer-assembled BYO-NOS model - SONiC or another NOS on merchant/Silicon One hardware, ECMP or Scheduled Ethernet, everything integrated and operated in-house - the hyperscaler operating model, and the continuum's honest lesson is that where an organization sits picks the pole: hyperscalers buy silicon a-la-carte and build the stack; most enterprises buy the stack and rent the expertise. What makes the BYO pole technically portable is SAI (Switch Abstraction Interface): SONiC holds intent in a Redis config store, orchestrator daemons translate it into SAI objects, and the ASIC vendor's SDK programs the silicon - so the NOS decouples from the ASIC vendor, and hardware lock-in drops to the silicon layer rather than the operating system. Two selection facts cut across the spectrum. Brownfield capability lives in the middle: the turnkey cloud-managed pod is greenfield-only (it manages its own switches, servers, BMCs, and NICs as one certified unit), and BYO-NOS builds are greenfield in practice - so an existing estate that must absorb AI pods selects the interoperable controller-plus-CVD model by constraint, not preference. And the turnkey pole's real product is prescriptive, validated tuning: the controller instantiates the RoCEv2 QoS (PFC/ECN thresholds) and load-balancing policy from certified templates - exactly the scarce expertise (Lossless fabric - PFC, ECN, DCQCN) that keeps enterprises off self-built lossless fabrics. At that pole you are buying the tuning as much as the tin.

Backend tenant overlays and the addressing plan

When the back-end is shared, the EVPN overlay choice is made per tenant, in two shapes (Overlays and BGP EVPN for the mechanics). A Layer 2 overlay puts a tenant's E-W NICs in one subnet on one L2 VNI - no gateway in the GPU path, flooding tamed by ARP suppression and Type-2 MAC/IP routes - the right shape for smaller workloads; its scale bound is the host, not the fabric: the GPU nodes' OS ARP cache must hold every peer, so size the broadcast domain to the ARP table rather than the VNI. A Layer 3 overlay (symmetric IRB with a per-tenant L3VNI, prefixes carried as Type-5) is what a tenant needs once its GPUs span subnets, and route leaking between tenant VRFs and a shared storage VRF is how every tenant reaches the common storage without collapsing isolation. On addressing, the doctrine splits cleanly. The underlay is static plumbing - private space, /30 links and /32 loopbacks - and because each back-end is an isolated island, the same underlay plan can be stamped identically onto every fabric the enterprise builds. The overlay is sized once, to the maximum GPU-endpoint count the cluster will ever reach: one master block counting one address per E-W NIC (per GPU on 1:1-NIC hosts, per GPU pair on 2:1 hosts - the NIC:GPU quantum again, Validated reference design - Cisco AI-ML lossless fabric (CVD)) plus gateway addresses wherever an L3 overlay routes, then carved into per-tenant subnets with growth headroom rather than leaning on EVPN's overlapping-address trick - overlap is legal but makes a dynamic multi-team estate harder to operate and troubleshoot. One placement note survives the plan: new scale units drop their GPUs anywhere in the fabric with L2/L3 reachability inside the tenant VRF, but a workload spanning old and new units inherits the spine-hop latency of that spread (Latency and the straggler problem).

Comparison

ai-operating-models-matrix

Recommendation / justification

Default to a dedicated, L3-routed, non-blocking island (BGP CLOS) for the GPU back-end, single-site, with SDN/ACI where central policy and multi-fabric integration justify it. Justify by isolation of the performance-critical RDMA traffic and the operational model the customer can run.

Hybrid transports across planes

Transport choice is per-plane, not whole-DC. A common AI data centre runs InfiniBand on the GPU back-end (lowest-latency compute interconnect) while Ethernet carries the storage, front-end, management, and client planes (cheaper, ubiquitous, ample at 400/800G), each scaled independently for cost and ROI (Fabric transport - Ethernet vs InfiniBand vs UEC). The two are joined by InfiniBand<->Ethernet gateways/bridges that handle protocol translation, and IP over InfiniBand (IPoIB) lets IP and socket-based legacy apps run over an IB fabric (IB's native addressing is not IP and has no sockets). As Enhanced/Scheduled Ethernet and UEC mature, scale IB deployments cautiously so you do not strand cost in a transport you may later migrate off.

The front end has its own unification fork. The stitched norm - EVPN-VXLAN inside the DC, L3VPN over SR-MPLS across the WAN, stitching gateways at the DCI - pays running costs: no traffic engineering inside the VXLAN domain, service chaining via PBR and VRF hand-offs, common vendor dependence on v4 underlay loopbacks, and a gateway tier to scale and troubleshoot. The alternative is end-to-end SRv6 with the compute or cloud-gateway node as the PE (FRR, VPP, or Cilium/eBPF host stacks): one data plane from workload to WAN, chaining and TE as path programming, no stitching tier - bought with newer multivendor edges and host data-plane engineering (Segment Routing).

What would change this (mid-scenario twist)

  • Central policy / multi-fabric integration needed -> SDN/ACI overlay.
  • Distributed training mandated -> multi-site with high-bandwidth DCI.

Validation checks

  • Is the back-end isolated and L3-routed (minimal L2), non-blocking?
  • Is the operating model (traditional vs SDN) matched to the team and integration needs?

IPv6 / dual-stack note

Underlay/overlay can be dual-stack; ensure routing, ECMP, and any EVPN carry both families for the fabric.

Spaced repetition

The usual connectivity model for the GPU back-end is [...].

The usual connectivity model for the GPU back-end is a dedicated, purpose-built island (separate from front-end/management/storage planes).

Why is the AI back-end L3-routed with minimal L2?

To avoid L2 scale/loop problems; reachability is propagated by BGP/IGP (EVPN only where an overlay is needed).

Traditional vs SDN for the fabric: traditional = [...]; SDN = [...].

Traditional vs SDN for the fabric: traditional = BGP-routed CLOS (eBGP underlay); SDN = controller/ACI for policy, automation, multi-fabric integration.

In a hybrid AI data centre, how are InfiniBand and Ethernet typically combined?

IB on the GPU back-end (lowest-latency compute interconnect); Ethernet on the storage, front-end, management, and client planes (cheaper, ubiquitous, 400/800G); joined by IB<->Ethernet gateways, with IPoIB carrying IP/socket apps over IB. Scale IB cautiously as Enhanced/Scheduled Ethernet and UEC mature.

The three AI-fabric operating models, and the key trade?

Self-managed NX-OS (box-by-box, most control and most toil); on-prem SDN controller (ACI / NDFC - central intent but you host the controller); cloud-managed fabric-as-a-service (e.g. Nexus HyperFabric - turnkey, controller in the vendor cloud, spans DC/colo/edge). The trade is control and in-house expertise vs deployment speed and dependence on vendor-cloud reachability.

In a cloud-managed (as-a-service) AI fabric, what happens to the running fabric if the controller is unreachable?

Nothing to in-flight collectives - the cloud controller schedules and configures but does not carry GPU traffic (the data plane is local); a controller outage affects change and visibility only, not the running RDMA data plane.

Converged fabric or dedicated GPU back-end - what decides?

Per-host GPU/NIC density: 1-2-GPU servers (inference/RAG/fine-tuning) can share one QoS-separated fabric for GPU, front-end and storage traffic; 8-GPU training nodes (8x400G of collective traffic each) force the dedicated lossless back-end, which is also an independent failure/change domain.

Which AI-fabric operating model can absorb an existing brownfield estate, and why?

The interoperable middle - private/controller-managed fabrics with CVDs - because turnkey cloud-managed pods are greenfield-only (they manage their own certified switch/server/NIC set as a unit) and BYO-NOS builds are greenfield in practice; brownfield selects the interoperable model by constraint, not preference.

The vendor-neutral API that lets SONiC program any merchant ASIC - decoupling the NOS from the silicon vendor - is [...].

The vendor-neutral API that lets SONiC program any merchant ASIC - decoupling the NOS from the silicon vendor - is SAI (Switch Abstraction Interface).

What bounds a single compute attachment domain, and why run several anyway?

The port budget - server-facing ports trade directly against uplink ports, so domain growth raises oversubscription toward the network - and domains double as facility fault domains, so multiple smaller domains per room bound environmental blast radius.

L2 or L3 overlay for a GPU tenant on a VXLAN EVPN back-end - what decides, and what bounds the L2 choice?

L2 (one subnet, one VNI, ARP suppression plus Type-2, no gateway in the path) suits smaller single-subnet workloads and is bounded by the GPU hosts' OS ARP cache, not the fabric; L3 (symmetric IRB, per-tenant L3VNI, Type-5) once the tenant's GPUs span subnets - with route leaking to a shared storage VRF for common storage.

How is the back-end addressing plan structured for a growing GPU cluster?

A static, reusable private underlay (/30s and /32s, stamped identically across isolated fabrics) plus one overlay master block sized to the maximum GPU-endpoint count - an address per E-W NIC (per GPU or per GPU pair by NIC ratio) plus gateways where L3 overlays route - carved into per-tenant subnets with headroom instead of overlapping VRF space.

In an L2 GPU-tenant overlay the real scale bound is the [...], not the fabric's VNI capacity.

In an L2 GPU-tenant overlay the real scale bound is the GPU hosts' ARP cache, not the fabric's VNI capacity.

Sources

  • Cisco AI POD for Enterprise Training and Fine-Tuning Design Guide (Jan 2026) - L2-vs-L3 tenant overlays, ARP-cache bound, storage-VRF leaking, underlay/overlay addressing doctrine; full 94-page pass 2026-07-26 (user-supplied PDF).
  • Cisco/NetApp/SUSE, FlexPod with SUSE Rancher for AI design guide (Dec 2023) - attachment-domain port-budget and facility-fault-domain sizing; full 75-page pass 2026-07-26 (user-supplied PDF).
  • Cisco Live BRKSP-2551 (2026), Introduction to Segment Routing - A foundation for Autonomous Networks (Donzelli, Cisco / Gorovoy, Nebius; incl. the Microsoft AI-fabric and Nebius DCI material)
  • Cisco/Hitachi Adaptive Solutions OCP AI + Cisco/NetApp FlexPod OpenShift AI MLOps CVDs (2024/2025) - OVN-Kubernetes/Geneve default CNI vs ACI CNI; design chapters passed 2026-07-25.
  • T. Pasanen, The Network Times (nwktimes.blogspot.com), SONiC ASIC-programming series (2026) - SAI pipeline, desired-vs-runtime state split; front-page post passed in full 2026-07-25 (DB dumps/CLI excluded by invariant).
  • Cisco Live BRKDCN-2921 - worked converged (2-GPU hosts) vs split (8-GPU hosts) designs.
  • Cisco Live BRKDCN-2677 (Cadora), AI Infra For The Rest of Us - operating-model continuum, greenfield/brownfield constraints, templated RoCEv2 tuning.
  • Cisco AI/ML data-center blueprint; eBGP-routed CLOS references.
  • Cisco U., "Hybrid Connectivity" (AI Transport) - IB+Ethernet coexistence, gateways/bridges, IPoIB.

domain: AI-Infra · blueprint-ref: AI-Infra 2.4/2.5/2.6/2.8 Infrastructure, L2, routing optimization, connectivity models · type: design-decision · status: complete · tags: [elective/ai-infra, ai/network, tradeoff/scale, tradeoff/operability]

Read more