Load balancing in AI fabrics - flow, flowlet, packet spray

AI backend traffic breaks the assumption enterprise ECMP relies on. Enterprise links carry many small flows that hash evenly; an AI backend carries a handful of massive elephant flows (GPU-to-GPU collectives at line rate, often Tb/s). Spreading those few flows well is a first-class design problem.

Why flow-based ECMP struggles here

Traditional L3 ECMP hashes each flow's 5-tuple to one uplink and pins all its packets to that path; it does not react to congestion. With only a few elephant flows, two or more can hash to the same uplink (hash collision / polarization), overloading one link while parallel links sit idle - triggering ECN/PFC and stalling barrier-synchronised jobs (Lossless fabric - PFC, ECN, DCQCN, Latency and the straggler problem). The richer the path set, the more a static hash wastes it.

There is a second, RoCEv2-specific reason the hash starves: the outer header barely varies. Backend GPUs sit in uniform subnets, every RoCEv2 packet carries the same UDP destination port (4791), and the one field left - the UDP source port - is assigned per Queue Pair (QP), so a GPU-pair conversation is a single immutable flow however much it carries. Two entropy levers follow, both fixes for static hashing and complementary to the adaptive modes below. Host-side: scale the QP count - the collective library can open many QPs per peer, splitting each conversation into many hashable flows (Meta reported up to ~40% training-throughput improvement from QP scaling). Switch-side: hash deeper than the 5-tuple - extend the ECMP hash into the RoCE header's destination-QP field (a UDF-style fixed-offset hash), so distinct QPs diverge even when the outer tuple is identical. Dual-stack angle: over IPv6, a host stack that populates the flow label per QP hands the switch a shallow hashable field that IPv4 simply does not have. One honesty check before buying the adaptive modes below: the entropy famine is at its worst when every GPU serves one job - the enterprise norm of many concurrent fine-tuning workloads restores entropy on its own - so measure the fabric's real flow mix before concluding that hash-based ECMP (plus the two levers above) cannot carry it.

The option spectrum (increasing balance, increasing requirements)

• Per-flow ECMP - simplest, congestion-blind; fine for many small flows, poor for few elephants.
• Static pinning - deterministically map given ingress ports/flows to chosen uplinks (manual or API). No reordering and fully predictable, but zero adaptivity - viable only for a few steady elephants or an undersubscribed fabric; the degenerate, deterministic end of the spectrum.
• Flowlet switching + adaptive routing - an RDMA transfer naturally arrives as flowlets (bursts separated by small gaps). The switch monitors link load and, when a path crosses a congestion threshold, moves subsequent flowlets to a less-congested path. The inter-flowlet gap keeps reordering bounded - the aging/gap threshold is sized to roughly 2x fabric RTT: shorter risks reordering (one burst split across paths still in flight), longer forfeits rebalancing and leaves elephants pinned. A strong middle ground - dynamic, but coarser than per-packet.
• Per-packet spraying - spread individual packets of one flow across all equal-cost paths for the most even load. The catch is reordering: sprayed packets arrive out of order, which classic in-order RDMA dislikes. It is enabled by reorder-tolerant transport - e.g. RDMA Write-Only with self-describing packets (each packet carries its own placement info), so the NIC can place data without strict ordering. Needs NIC and fabric support.

Who controls the spray - the Ethernet-mode taxonomy

The spectrum above can be re-read along one design axis: where the intelligence that sprays and re-orders packets lives. Cisco Silicon One fabrics expose this as four escalating modes, and that fabric-controlled vs endpoint-controlled axis is what actually changes a design answer. Stateless ECMP Ethernet is the classic 5-tuple hash - no load awareness, the polarization problem above. Enhanced Ethernet lets the switch weigh real-time link load when balancing, so it dodges the worst collisions, but by default it does not re-order within a flow (coarser than per-packet) and only reaches true spray-and-re-order with SmartNIC help at the endpoints. Ultra Ethernet (UEC) puts the intelligence at the host edge - the endpoint NIC drives adaptive packet spraying (Fabric transport - Ethernet vs InfiniBand vs UEC). Scheduled Ethernet has the fabric do it all: the ingress leaf sprays per-packet by fabric load and the egress leaf re-orders each flow before delivery, so hosts receive in-order RDMA with no endpoint changes - giving deterministic, traffic-independent balancing, the closest practical approach to a genuinely non-blocking Clos (Rail-optimized CLOS topology).

Why it matters: a static hash leaves parallel uplinks idle, and the hot link that results triggers incast (many senders overflow one shared egress port) and head-of-line (HOL) blocking (a packet stuck at the front of a queue stalls everything behind it even when those packets could move), both of which stall barrier-synchronised collectives (Latency and the straggler problem). Adaptive and sprayed modes keep every link busy so neither becomes the straggler. The scenario question is therefore how much endpoint capability you have: with no NIC freedom, fabric-controlled Scheduled Ethernet still delivers deterministic balancing; with capable UET or SmartNIC endpoints, Ultra or Enhanced spray-and-re-order is on the table.

One more point on the axis deserves a name: vendor closed-loop adaptive routing (NVIDIA Spectrum-X). Here the endpoint SuperNIC and the switch cooperate - the fabric shares congestion state and the BlueField-3 makes routing decisions against it - a tightly coupled NIC+switch system rather than either end acting alone. The design-relevant fact from the AI POD CVD: this mode is validated beyond NVIDIA's own switches onto Nexus fabrics as a licensed add-on, so an NVIDIA-endpoint estate can get closed-loop adaptive routing without leaving a Cisco fabric - ecosystem coupling and license cost traded for the tightest congestion loop on the menu.

The standards answer to the same closed loop is taking shape inside UEC: the specification standardises in-band telemetry - real-time congestion state embedded in the packet headers themselves - hardware path-measurement tables at nanosecond precision driving per-packet forwarding decisions on queue occupancy and path latency, and a defined NIC-to-switch interaction model for path selection and congestion feedback: the control plane of adaptive routing as an open interface rather than a vendor pairing (Fabric transport - Ethernet vs InfiniBand vs UEC). The AI POD CVD's positioning is the practical read: DLB is the recommended default today, flowlet and vendor adaptive routing are the alternates for existing estates, and UEC balancing is what to plan for once production-ready - the vendor-neutral escape from the licence-and-ecosystem coupling above. One certification nuance rides along: NVIDIA's Enterprise-RA endorsement currently requires the licensed Spectrum-X enablement active on the east-west fabric (and adherence to its patterns, e.g. the 4-leaf, 2-rail-per-switch HGX design), so a plain DLB or UEC Nexus fabric can be a perfectly sound AI POD yet not an ERA-badged one - a procurement distinction, not a performance one.

A note on how Scheduled Ethernet reaches that determinism: the fabric uses ingress virtual output queuing (VOQ) - each ingress port queues per destination output/class and a scheduler grants fabric access, so the whole leaf-spine-leaf fabric behaves as one logical switch with drops confined to the ingress leaf (not the spine or egress leaf). That is the mechanism behind its high, traffic-independent link utilisation - at the cost of tighter coupling between elements and a smaller ecosystem than standard ECMP Ethernet.

One more pole completes the axis: explicit source routing. With SRv6 uSID the sender - or its controller/scheduler - encodes plane and path in the packet's destination address, so placement is exact per flow or per QP with no hash anywhere in the fabric: deterministic like static pinning, but programmable - the scheduler re-plans placement per job or per step instead of an operator re-pinning ports. A hyperscale AI back-end runs precisely this - uSID on SONiC for deterministic load balancing across its four fabric planes - proving the pole at production scale. The trade mirrors its position on the axis: all path intelligence lives at the source/controller, the fabric contributes no adaptivity, and congestion response is only as fast as the re-planning loop (Segment Routing).

Cisco knob

Nexus Dynamic Load Balancing (DLB) implements both flowlet and per-packet (spray) modes at Layer 3 - the platform expression of the spectrum above.

Recommendation / justification

Match the method to flow profile and endpoint support: with reorder-tolerant NICs and a capable fabric, per-packet spray makes the most even use of a non-blocking Clos; otherwise flowlet + adaptive routing is the safe default; plain ECMP only where flows are many and small. The aim is to keep every parallel uplink busy so a single hot link never becomes the straggler that gates the job.

What would change this (mid-scenario twist)

• NICs/transport can't tolerate reordering -> stay on flowlet + adaptive, not per-packet.
• Mostly MoE All-to-All -> even spraying matters most; bias toward per-packet where supported.
• Storage/front-end mixed in -> keep its many-small-flows on ECMP; isolate the elephant collectives (Validated reference design - Cisco AI-ML lossless fabric (CVD)).

IPv6 / dual-stack note

Whatever the method, ensure hashing/spray entropy and adaptive-routing telemetry behave identically for IPv4 and IPv6 so balance does not differ by family.

Related

• Rail-optimized CLOS topology
• Lossless fabric - PFC, ECN, DCQCN
• Latency and the straggler problem
• Parallelism strategies and collective traffic patterns

Spaced repetition

Sources

• Cisco AI POD for Enterprise Training and Fine-Tuning Design Guide (Jan-2026 revision): single-job-vs-concurrency entropy observation.
• Cisco Live BRKSP-2551 (2026), Introduction to Segment Routing - A foundation for Autonomous Networks (Donzelli, Cisco / Gorovoy, Nebius; incl. the Microsoft AI-fabric and Nebius DCI material)
• Cisco AI POD for Enterprise Training and Fine-Tuning Design Guide (CVD, Jan 2026) - Spectrum-X closed-loop adaptive routing on Nexus (licensed); UEC mechanism triad, DLB-today/UEC-next positioning, ERA licensing gate; full 94-page pass 2026-07-26 (user-supplied PDF)
• Cisco Live BRKDCN-2921 (AI DC network best practices) - RoCEv2 entropy, UDF/destination-QP hashing, DLB modes; BRKDCN-2677 - QP scaling (Meta figure).
• Toni Pasanen, Deep Learning for Network Engineers (2025), Ch 12 (Flow, Flowlet, and Packet-Based Load Balancing).
• Cisco U., "Nonblocking Lossless Fabric" (Key Network Challenges and Requirements for AI Workloads) - Silicon One Ethernet modes.

domain: AI Infrastructure · type: design-note · status: complete · tags: · source: Pasanen, "Deep Learning for Network Engineers" (2025), Ch 12