AI: Network
In one line RDMA lets a NIC read/write remote memory directly - bypassing the CPU and kernel - which is what makes GPU-to-GPU communication fast enough for training; RoCE carries RDMA over Ethernet. RDMA Zero-copy, kernel-bypass, NIC-offloaded transfers via the verbs API: microsecond latency, line-rate
AI: Network
The signature AI-Infra design decision: what transport carries the backend / scale-out GPU fabric (the east-west, RDMA, highest-speed network that GPUs use for collective communication). Get this wrong and every other choice inherits the pain. Requirement / business driver Move synchronized collective traffic (all-reduce / reduce-scatter) between
AI: Network
In one line The set of mechanisms that make an Ethernet fabric near-lossless so RoCEv2 RDMA can run without the retransmits that would wreck Job Completion Time. How it works * ECN (Explicit Congestion Notification) - end-to-end, early. A congested switch marks the IP ECN bits; the receiver returns
AI: Network
Requirement / business driver Build a non-blocking GPU back-end fabric whose topology minimises hops and contention for collective communication - because the topology directly sets training performance. CLOS / spine-leaf baseline Leaf (ToR) + spine, non-blocking via ECMP, scaled by adding spines and pods. Every leaf reaches every other through
AI: Network
Requirement / business driver Interconnect GPUs, leaves, and spines at 400/800G reliably and affordably - at AI scale, optics are a major cost, power, and reliability factor (tens of thousands of links). Options * DAC (passive copper) - ~1-3 m, cheapest, negligible power - intra-rack. * AEC (active copper) - ~to 7 m - short
AI: Network
In one line Distributed training is barrier-synchronised, so the slowest path or GPU (the straggler) gates the entire step - which is why tail latency, not average, governs Job Completion Time (JCT). Why one slow link stalls thousands of GPUs Collectives (all-reduce, all-gather) require every GPU to finish
AI: Network
Requirement / business driver Size the back-end fabric so it never throttles the GPUs - because idle GPUs waiting on the network waste the most expensive resource in the building. Per-GPU bandwidth and non-blocking Modern training GPUs need 400G (moving to 800G) per GPU on the back-end, and
AI: Network
In one line Precise time sync (PTP / IEEE 1588) gives the fabric a common, sub-microsecond clock - essential for correlating telemetry, measuring congestion, and coordinating distributed workloads. PTP basics Hardware timestamping, a grandmaster clock, and boundary / transparent clocks in the path deliver sub-microsecond accuracy - versus NTP's millisecond
AI: Network
Requirement / business driver Decide how the AI fabric is structured and operated - a dedicated island vs part of the DC, traditional routing vs SDN, single vs multi-site. Connectivity models (2.8) * Isolated / purpose-built island - a dedicated back-end fabric for GPU/RDMA traffic, separate from front-end, management,
AI: Network
A concrete, tested anchor for the abstract fabric notes: Cisco's Validated Design for a non-blocking, low-latency, lossless Ethernet fabric carrying RoCEv2 for AI/ML. It pins the topology math, the eBGP underlay, the server-attach model, and the QoS recipe (Lossless fabric - PFC, ECN, DCQCN) into
AI: Network
The bridge between the workload and the fabric: how a model is split across GPUs decides the traffic the network must carry. A network architect doesn't train the model, but must know which parallelism strategy and which collective operations produce the east-west load (AI-ML workloads and
AI: Network
AI backend traffic breaks the assumption enterprise ECMP relies on. Enterprise links carry many small flows that hash evenly; an AI backend carries a handful of massive elephant flows (GPU-to-GPU collectives at line rate, often Tb/s). Spreading those few flows well is a first-class design problem.