Validated reference design - Cisco AI/ML lossless fabric (CVD)

A concrete, tested anchor for the abstract fabric notes: Cisco's Validated Design for a non-blocking, low-latency, lossless Ethernet fabric carrying RoCEv2 for AI/ML. It pins the topology math, the eBGP underlay, the server-attach model, and the QoS recipe (Lossless fabric - PFC, ECN, DCQCN) into one worked example. Treat the numbers as one validated reference point, not the only answer.

The non-blocking principle (the load-bearing rule)

A fabric is non-blocking when each leaf's uplink bandwidth to the spine equals its front-panel (server-facing) bandwidth - no oversubscription. Every other choice serves this rule.

Worked example: 128 servers x 2 x 100G = 256 x 100G server ports. Each leaf offers ~28 x 100G host ports + 8 x 400G uplinks (uplink BW = host BW -> non-blocking); 10 leaves cover the ports. The design then chooses four spines, not two - because scale and resiliency dominate for AI/ML - giving each leaf 8 x 400G (~3.2 Tbps) to the spine and leaving spare spine ports to grow without breaking the non-blocking property. Capacity method in Bandwidth and capacity planning; topology in Rail-optimized CLOS topology.

Server attachment: routed access ("L3 to the leaf")

Each server is dual-homed to two different leaves at 100G for redundancy and bandwidth. The notable choice: each NIC lands on an untagged access port in its own subnet, and the leaf exposes a /24 SVI advertised by BGP - so a NIC needs only an IP and a default route to its leaf SVI. No port-channel / LACP / NIC bonding is required:

• Outbound load-balancing is done by the host OS; inbound spreads via DNS round-robin.
• This simplifies server/NIC operations - but the hosts still must enable ECN and PFC (lossless is not network-only).

  A connectivity-model trade-off versus L2/bonded access - see Connectivity models and SDN in AI fabrics.

eBGP underlay (control plane)

• eBGP spine-leaf, with the leaves sharing one ASN and the spines another. ECMP across the four spines via maximum-paths (matching the four equal-cost uplinks).
• Leaves redistribute their connected loopback + server-SVI prefixes (selected by a route tag) into BGP - no IGP needed.
• BFD on the uplinks catches soft failures fast (hard failures are caught by link-down).
• Because the leaves reuse one ASN, the design defeats eBGP's AS-path loop check with allowas-in on the leaf and disable-peer-as-check on the spine - the standard shared-leaf-ASN technique. (BGP design in BGP in enterprise design.)

Building and operating it

The fabric - including the PFC/ECN QoS - is built from an eBGP fabric template in NDFC (Nexus Dashboard Fabric Controller), which discovers switches by CDP from a seed and pushes consistent fabric-wide QoS; some QoS is supplied as freeform CLI. The same controller adds leaves/spines and works whether the design is L3-to-leaf or a VXLAN overlay. This is the template-driven, consistency-first operating model behind AI-assisted operations - AIOps.

Front-end vs back-end networks (and a 400G back-end reference)

Large GPU deployments often split into two fabrics with different jobs:

• Front-end - the shared/general network (north-south, storage access, management, in-band ops); may be the same lossless fabric or a separate one.
• Back-end - a dedicated GPU-to-GPU fabric for GPUDirect RDMA (and GPUDirect Storage), carrying only collective traffic and typically run at 400G. Same RoCEv2 lossless requirements as everything here, just sized for 400G hosts.

400G back-end reference (blueprint): 512 GPUs at 8/server = 64 servers = 512 x 400G NICs. Leaf = a 64x400G-class switch giving 32 host ports + 32 uplinks per leaf -> 16 leaves; 8 spines, each leaf 4 x 400G to every spine -> non-blocking. (A 1024-GPU build scales to 32 leaves / 16 spines with 2 links each.)

The AI POD front end: services, sizing, and the management plane

The same CVD works the front-end fabric into doctrine rather than an afterthought. Because the back-end is a sealed island, the FE is the cluster's only gateway: it carries PXE boot, DNS/DHCP, workload orchestration, storage access (including GPUDirect Storage), user and inference traffic - and the out-of-band channel NCCL/RCCL themselves use for bootstrap and control. It is the same two-tier spine-leaf under the same Nexus Dashboard as the back-end, but its sizing rule inverts: start at ~4:1 oversubscription and let flow telemetry, not doctrine, drive uplink additions (Bandwidth and capacity planning). Host attachment is a fork: a single trunked FE NIC for everything needs bonding for HA and drags in real caveats - a BlueField DPU must run in NIC mode to bond with LACP, and PXE must obtain an address and load the OS before the LACP bond forms or the boot fails - versus a dedicated storage NIC, the stronger call once inference joins training on the node (a C845A cannot take it: one FE NIC only). Because the FE mixes lossless RoCEv2 storage with best-effort everything else, a PFC lane for the storage class is mandatory, and at scale (or with SDS back ends) the storage systems move onto their own storage-leaf pair so storage I/O neither contends with compute-leaf traffic nor scales in lockstep with it. Operations then split across two networks: a physically separate OOB pair (1/10G) owns bring-up and last-resort access - switch management, server CIMC, even the BlueField management ports, plus AD/DHCP/PXE/DNS reachability - while in-band management rides the FE fabric and is not optional for one thing: per-flow hardware telemetry is only feasible in-band, because the streams outrun a 1/10G OOB plane - so day-2 flow analytics is an FE design dependency, not a monitoring add-on. Finally the control planes get their own home: a dedicated management and services cluster of general-purpose UCS nodes on the FE fabric, sized by the stack's HA minima (two BCME head nodes, plus two for SLURM or three for a Kubernetes/OpenShift control plane - compact and hosted-control-plane variants included, with virtualised AD/DNS/DHCP alongside) - attached to the compute leaves when small, to dedicated management/services leaves when grown, which is also the landing zone for later security services (Protecting AI services - WAF, API security, and inference endpoints).

The enterprise band: NVIDIA Enterprise RA and scalable units

The references above start at hundreds of GPUs; the enterprise band has its own named pattern. The NVIDIA Enterprise Reference Architecture (ERA) - an NVIDIA design certified with specific vendor switching - covers 4 to 128 HGX nodes built from 4-node Scalable Units (SUs): the SU is the growth granule (32 GPUs at 8 per node), sized for the enterprise workload span from large inferencing through fine-tuning to moderate training. Its one structural threshold is the spine split: up to ~12 nodes a single combined spine carries CPU, storage, management, and GPU traffic; at 16 nodes the design separates a north-south spine (CPU/storage/management) from an east-west GPU spine - the front/back-end split arriving as a scale trigger rather than a day-one axiom. Two operational facts travel with the pattern. An HGX-class node presents 20+ connections across three logical networks at three speeds (400G-per-GPU backend, frontend, BMC/management), so at this band the cable plan and bill of materials are themselves failure surfaces - which is why validated templates that auto-generate topology, cabling, and an orderable BOM carry genuine design value (Rail-optimized CLOS topology, Optics and cabling selection). And "certified" means the switch + server + NIC combination was validated as a unit, shrinking the integration risk that dominates enterprise AI builds.

The two NVIDIA RA programs: scale bands and postures

The enterprise band above is one of two published RA programs, and the split itself is a design lesson. The Enterprise RA covers 32 to 256 GPUs (4 to 32 nodes): single-tenant, Ethernet-based, derived from the larger programme but right-sized, and deliberately adaptable - partners may keep the node patterns and substitute their own networking. The NCP (Cloud Partner) RA starts at 128 nodes and scales past 16,000 GPUs, with supercluster lineage to ~100,000: foundational-model training and GPU-as-a-service, single- and multi-tenant, InfiniBand or Ethernet, and deliberately rigid - it is deployed by the vendor's own services, and the rigidity is the feature: no per-site re-engineering, every deployment identical. The NCP design also fixes the plane taxonomy by name: TAN (tenant access, north-south - with storage as its primary consumer), SMN (out-of-band secure management), CIN (the GPU cluster interconnect, east-west), and NVLink (the per-rack scale-up domain) - with the rule that TAN and SMN are always Ethernet while only the CIN carries the Ethernet-or-InfiniBand fork. Its physical grain is the POD: GPU PODs of up to 1,152 GPUs (up to 64 of them) plus one Core POD holding everything that is not accelerated compute - control-plane nodes, general-purpose compute, the high-speed storage cluster, a utility cluster that bootstraps the data centre, and a DC-edge cluster (the outside-world interface, firewalls included).

The Cisco AI POD scale-unit model (and the two scale-beyond forks)

Cisco's AI POD design (the training/fine-tuning CVD) sharpens the ERA pattern in one structural way: the scale unit packages the leaf pair with the compute nodes, so every growth increment is a self-contained compute+network block and the leaf port-class becomes the cluster quantum. A 32x400G leaf pair, a 64x400G pair, or a 64x800G Silicon One pair each fix how many GPU nodes one unit carries - and, multiplied out, the published two-tier maxima: about 512 GPUs on a two-spine fabric for the largest unit type, doubling to about 1024 on four spines - at the price that adding a spine pair means re-cabling every leaf's uplinks across all spines. The same tables expose a second quantum: hosts that share one E-W NIC between two GPUs (200G per GPU) double the GPUs a leaf pair carries versus one 400G NIC per GPU - per-GPU east-west bandwidth traded directly for cluster density, a NIC:GPU-ratio decision made at server selection (Bandwidth and capacity planning). Past the two-tier ceiling the design names two forks. Replicate isolated fabrics: identical islands, simplest and fully non-blocking, but a job cannot span islands and spare capacity strands per island - right only while the largest workload fits one island (Connectivity models and SDN in AI fabrics). Add the third (super-spine) tier: the whole GPU pool becomes one schedulable domain at the cost of an extra hop's latency - and it is only cheap if spine downlink ports were reserved on day one, which itself lowers the two-tier maximum. The fork is decided by the sizing doctrine: island size versus the single largest workload (Cost, ROI, and scalability of AI infrastructure). One more day-one stance from the same CVD: even a single starter scale unit ships as a 2-spine, 2-leaf fabric - a spine layer from the first four nodes - because its upfront cost is minimal against the redesign and downtime of retrofitting one later, and the leaf-add growth path then never changes shape.

Latency budget and scaling

Per-switch latency on these platforms is about 1.5 us, so a leaf-spine-leaf path is roughly ~4.5 us end-to-end - and latency-sensitive peers placed under the same leaf pair skip the spine hop entirely (Latency and the straggler problem). Scale in this order: add leaves (spare spine ports preserve non-blocking) -> bigger spines (e.g. 64x400G) -> a three-tier "super-spine" interconnecting multiple non-blocking pods. The dedicated build follows MSDC principles with BGP to the L3 leaf; where multiple tenants/functions share it, overlay MP-BGP EVPN VXLAN (Connectivity models and SDN in AI fabrics) for separation.

Even a non-blocking fabric still needs congestion management: incast - two senders at line rate to one receiver - congests the egress port regardless, so ECN+PFC (Lossless fabric - PFC, ECN, DCQCN) stay mandatory. A good (non-blocking) design just means the congestion-control algorithms work less, so jobs finish faster.

What would change this (mid-scenario twist)

• More hosts -> add leaves (spare spine ports preserve non-blocking); past spine capacity -> scale the spine or go multi-plane/rail (Rail-optimized CLOS topology).
• Strict multi-tenant isolation -> a VXLAN EVPN overlay instead of plain L3-to-leaf (Connectivity models and SDN in AI fabrics).
• Ultra-low-jitter / very large collectives -> reconsider InfiniBand vs Ethernet (Fabric transport - Ethernet vs InfiniBand vs UEC).

Validation checks

• Confirm uplink BW == host BW on every leaf (the non-blocking invariant).
• Confirm ECMP is actually spreading (maximum-paths effective; flows hashed across all spines).
• Confirm ECN marks before PFC pauses, CNP is protected, and the PFC watchdog is armed (Lossless fabric - PFC, ECN, DCQCN).
• Confirm hosts have ECN + PFC enabled - the design fails silently if they don't.

Spaced repetition

Sources

• Cisco AI POD for Enterprise Training and Fine-Tuning Design Guide (Jan 2026): spine-from-day-one starter doctrine; C845A scale-unit density tables; front-end and management-plane doctrine - full 94-page pass 2026-07-26 (user-supplied PDF, lifts the batch-62 fetch cap).
• NVIDIA, Enterprise Reference Architecture Overview white paper, Introduction (fetched in full 2026-07-25) + NCP Software Reference Guide, Data Center Architecture (fetched in full 2026-07-25) - two-programme taxonomy, TAN/SMN/CIN/NVLink planes, POD/Core-POD grain.
• Cisco AI POD for Enterprise Training and Fine-Tuning Design Guide (CVD, Jan 2026) - scale-unit model, two-tier maxima, islands-vs-super-spine fork
• Cisco, Validated Design for Data Center Networking Blueprint for AI/ML Applications (Jun 2023; NX-OS 10.3.2.F, NDFC 12.1.2) and the companion DCN Blueprint for AI/ML.
• Cisco Live BRKDCN-2677 (Cadora) - NVIDIA Enterprise RA, scalable units, spine-split threshold, server network density.

domain: AI Infrastructure · type: design-note · status: complete · tags: · source: Cisco Validated Design (DCN Blueprint for AI/ML), Jun 2023