AI: Network

Rail-optimized CLOS topology

Miro Kosut

Miro Kosut

— 8 min read
Share

Requirement / business driver

Build a non-blocking GPU back-end fabric whose topology minimises hops and contention for collective communication - because the topology directly sets training performance.

CLOS / spine-leaf baseline

Leaf (ToR) + spine, non-blocking via ECMP, scaled by adding spines and pods. Every leaf reaches every other through the spine in a fixed, predictable hop count.

Rail-optimized design

In a multi-GPU server, GPU0 of every server connects to the same "rail" (leaf), GPU1 to rail-1, and so on. Because collectives are rank-aligned (all-reduce exchanges between same-rank GPUs), most traffic stays within a rail - leaf-local or one hop - rather than crossing the spine. Result: fewer hops, less cross-fabric contention, lower tail latency (Latency and the straggler problem). Rails pair with intra-server scale-up (NVLink) for the GPUs inside a box (Scale-up vs scale-out).

Topology variants

  • Fat-tree (non-blocking Clos) - the standard backend: a folded Clos sized so uplink BW = host BW; scale spine capacity by adding spines, no redesign (Validated reference design - Cisco AI-ML lossless fabric (CVD)). "Two-tier" describes the physical layers (leaf+spine); "three-stage" describes the logical path (leaf-spine-leaf) - the same fabric.
  • Rail-optimized Clos - GPUs of the same rail (same NIC index across hosts) home to the same leaf, with a spine for cross-rail traffic; gives consistent per-GPU paths.
  • Rail-only (L2) - rail leaves with no spine and no cross-rail links; cross-rail GPU-to-GPU is handled inside the host over NVLink (hop to the GPU whose NIC sits on the right rail). Saves the spine tier but leans on intra-host NVLink and tight GPU placement.

    Balancing traffic across whichever topology is its own problem - see Load balancing in AI fabrics - flow, flowlet, packet spray.

Scale

Rail leaves connect up to spines (and pods/super-pods) for the scale-out dimension; the back-end is sized non-blocking (1:1) on the GPU rails (Bandwidth and capacity planning).

Recommendation / justification

Use a rail-optimized non-blocking CLOS for the GPU back-end so collective traffic is rank-aligned and contention-free; reserve oversubscription for front-end/storage planes. Justify by collective traffic patterns and JCT, not generic DC design. Pair with good load balancing (Load balancing and hashing - ECMP and entropy) so flows actually spread.

Backend build: NIC ports, rails per switch, and pods

Above the topology sits the physical build of the GPU back-end. NIC per GPU: a non-blocking back-end gives each GPU its own NIC port (dedicated) rather than sharing one NIC across GPUs - a shared NIC oversubscribes the GPU's path and defeats the 1:1 rail. NIC ports: a single-port NIC puts each GPU on one rail (simplest, cheapest); a dual-port NIC gives each GPU two rails, doubling per-GPU bandwidth and surviving a NIC/link/leaf failure. Rails per switch is the consolidation knob: single-rail-per-switch dedicates a switch to one rail (most switches and cabling, but the smallest failure domain - losing a switch costs one rail), while multi-rail-per-switch terminates many rails on one switch (fewest switches, least cabling, but a switch loss takes many rails with it); dual-rail sits between. Scaling units: GPUs group into a host-segment, host-segments into a Pod (a self-contained non-blocking unit - e.g. 64 GPUs on a leaf-spine), and Pods link through a higher spine into a Multi-Pod / super-pod fabric - so you grow by adding pods rather than redesigning (Bandwidth and capacity planning). The pod is also the natural failure and maintenance boundary (Fabric resiliency and failure handling).

The cross-rail escape path: PXN and its asymmetry

Rail discipline depends on a quiet helper: PXN (PCIe/NVLink proxy), by which the collective library moves a message inside the sending host - across NVLink to the GPU whose NIC sits on the destination's rail - before injection, so inter-node traffic stays on one rail end-to-end. The asymmetry: receive-side PXN is not implemented in NCCL/RCCL, so the rank swap happens only at the origin, and the library assumes rail reachability from the presented topology without validating it. If rail k loses connectivity to the remote GPU, traffic addressed to it is dropped, not re-railed through a healthy rail plus a host-side hop at the far end. That single behaviour is much of why rail-only remains experimental: a plane failure becomes a hard job fault rather than a degraded path. The pragmatic middle keeps a leaf-spine layer above the rails - even deliberately oversubscribed - purely as the cross-rail escape path (Fabric resiliency and failure handling).

Rail groups and the 2/4/8-way ladder (enterprise band)

The rails-per-switch knob above has a named enterprise expression in the Cisco AI POD design. For an 8-GPU, 8-NIC node the ladder runs 2-way (two leaves, four rails each - the shape a scale unit's leaf pair produces by construction), 4-way (four leaves, two rails each, spanning two scale units) and 8-way (a dedicated leaf per rail - the classic large-cluster form). The rails terminating on one switch form a rail group, and the group is the point of the ladder: traffic between any rails of the same group is single-hop - including cross-rail traffic - with no spine transit and no PXN, so a 2-way build gives cheap cross-rank reach that an 8-way build must buy through the spine or the NVLink proxy. What 8-way buys back is rail capacity: with a whole switch per rail, the rail size equals the switch's downlink port count once up to half the ports are reserved for non-blocking uplinks - the arithmetic that fixes how many same-rank GPUs (nodes) one rail carries. Its cost is physical: every node needs any-to-any reach to all eight rail switches, so the rack layout and cable plan are designed around the rails rather than the reverse (Optics and cabling selection). The enterprise default then falls out naturally: the 2-way group is what a leaf-pair scale unit already is - row-local cabling, cross-rail reach for free - and it suits the many-concurrent-workload profile; 8-way is bought when one very large rank-aligned job dominates and its all-reduce must own the lowest same-rank path.

Hyperscale tiering: zones, buffer asymmetry, and the scheduler

Meta's production pattern shows what the pod doctrine becomes at building scale. The unit is an AI Zone: a two-stage Clos in which rack-level rail switches (in-rack copper DAC to the GPUs) uplink over 400G single-mode fibre to a modular, deep-buffered spine whose buffer is statically carved per port - a deliberate buffer asymmetry where the shallow, cheap leaf tier stays low-latency while the deep spine absorbs collective bursts (the reason persistent spine->leaf PFC has not been seen in years of production, Lossless fabric - PFC, ECN, DCQCN). LLM-scale jobs outgrow one zone, so an aggregation tier joins the zone spines and stretches the routable RoCE domain across the building - and here the doctrine inverts: the cross-zone links are oversubscribed by design, and the gap is closed not with more optics but with a topology-aware job scheduler that places ranks by a minimum cut of the collective's traffic so most bytes never leave their zone (Parallelism strategies and collective traffic patterns). Above the pod, non-blocking stops being an axiom and becomes a cost trade solved jointly by the network and the scheduler (Bandwidth and capacity planning). The empirical anchors are worth quoting in a design defence: static per-slice path pinning collapsed under fragmented job placement and failure re-hashing (training degraded by over 30%), and the interim fix - doubling rail-switch uplinks to run them 1:2 under-subscribed - bought headroom at twice the network cost until entropy-based balancing matured (Load balancing in AI fabrics - flow, flowlet, packet spray).

Comparison

rail-switch-design-matrix

What would change this (mid-scenario twist)

  • Larger GPU count -> add spines/pods while keeping rails non-blocking.
  • Severe hash polarisation on rails -> adaptive/flowlet load balancing.

Validation checks

  • Are the GPU rails non-blocking (1:1) and rank-aligned?
  • Does the spine layer preserve non-blocking as pods scale?

IPv6 / dual-stack note

Underlay routing (usually eBGP) can be v4 or v6; keep ECMP entropy working for whichever family carries RoCEv2.

Spaced repetition

What is "rail-optimized" about a rail-optimized CLOS?

Same-rank GPUs across all servers attach to the same leaf/rail, so rank-aligned collective traffic stays leaf-local (few hops), cutting contention and tail latency.

Why is the GPU back-end fabric built non-blocking (1:1)?

Collectives saturate links; any oversubscription becomes congestion and a straggler that stalls the whole synchronized step.

Rails handle the [...] dimension; [...] handles GPUs inside one server.

Rails handle the scale-out (inter-server) dimension; NVLink/scale-up handles GPUs inside one server.

Rail-optimized vs rail-only topology?

Rail-optimized = rail leaves plus a spine for cross-rail traffic (consistent per-GPU paths); rail-only = rail leaves with no spine, cross-rail handled inside the host over NVLink - saves the spine tier but depends on NVLink and placement.

Single-rail-per-switch vs multi-rail-per-switch in a GPU back-end - the tradeoff?

Single-rail-per-switch dedicates a switch to one rail: most switches and cabling but the smallest failure domain (a switch loss costs one rail). Multi-rail-per-switch consolidates many rails on one switch: fewest switches and cabling but a switch loss takes many rails. Dual-rail is between.

Why give each GPU a dedicated NIC port - and when a dual-port NIC?

Dedicated NIC-per-GPU preserves the non-blocking 1:1 rail (a shared NIC oversubscribes the GPU path); a dual-port NIC adds a second rail for double per-GPU bandwidth and survival of a NIC/link/leaf failure.

How does a pod-based back-end scale?

GPUs form a host-segment, host-segments form a Pod (a self-contained non-blocking unit, e.g. 64 GPUs), and Pods link via a higher spine into a Multi-Pod/super-pod - you grow by adding pods, and the pod is the failure/maintenance boundary.

At hyperscale, how is the oversubscribed cross-zone tier made to work?

A topology-aware job scheduler places ranks by a minimum cut of the collective's traffic so most bytes stay inside a zone - above the pod, non-blocking becomes a cost trade solved jointly by network and scheduler.

In Meta's AI Zone design, which tier is deep-buffered and why?

The modular spine (buffers statically carved per port) - it absorbs collective bursts so the shallow low-latency leaves rarely see persistent PFC; a deliberate buffer asymmetry between tiers.

Rail-only stays experimental partly because receive-side PXN is unimplemented - a rail that loses reachability to the remote GPU sees traffic [...], turning a plane failure into a job fault; rail-optimised designs therefore keep (even oversubscribed) cross-rail spine capacity as the escape path.

Rail-only stays experimental partly because receive-side PXN is unimplemented - a rail that loses reachability to the remote GPU sees traffic dropped rather than re-railed, turning a plane failure into a job fault; rail-optimised designs therefore keep (even oversubscribed) cross-rail spine capacity as the escape path.

What is a rail group, and why does it matter in 2-way and 4-way rail designs?

The set of rails terminating on one leaf - traffic between any rails of the same group is single-hop (even cross-rail) with no spine transit or PXN; consolidating rails per switch trades rail capacity and failure domain for cheap cross-rail reach.

Why does the AI POD default to a 2-way rail-optimised build?

A scale unit's leaf pair is a 2-way rail group by construction - four rails per switch, cross-rail single-hop inside the group, row-local cabling - which fits many concurrent enterprise workloads; 8-way is reserved for one dominant rank-aligned job that must own the lowest same-rank path.

In an 8-way rail design the [...] after reserving up to half the ports as non-blocking uplinks.

In an 8-way rail design the rail size equals the leaf's downlink port count after reserving up to half the ports as non-blocking uplinks.

Sources

  • Cisco AI POD for Enterprise Training and Fine-Tuning Design Guide (Jan 2026) - rail groups, 2/4/8-way ladder, rail-size arithmetic, cabling consequence; full 94-page pass 2026-07-26 (user-supplied PDF).
  • P. Gervasi, networkphil.com, Rail-Optimized Networking for AI Training Workloads (Apr 2026, incl. comment-thread PXN detail, T. Conrad) - full-text pass 2026-07-25.
  • Meta Engineering, RoCE Networks for Distributed AI Training at Scale (2024) - AI Zone tiering, cross-zone oversubscription, scheduler minimum-cut, path-pinning failure; full-text pass 2026-07-25.
  • Cisco AI/ML data-center blueprint; Nvidia rail-optimized reference designs.

domain: AI-Infra · blueprint-ref: AI-Infra 2.8 Connectivity models (CLOS) / topology · type: design-decision · status: complete · tags: [elective/ai-infra, ai/network, tradeoff/scale, tradeoff/latency]

Read more