AI: Network

Latency and the straggler problem

Miro Kosut

Miro Kosut

— 2 min read
Share

In one line

Distributed training is barrier-synchronised, so the slowest path or GPU (the straggler) gates the entire step - which is why tail latency, not average, governs Job Completion Time (JCT).

Collectives (all-reduce, all-gather) require every GPU to finish before the next step begins. A single congested link, dropped packet, or slow NIC makes thousands of GPUs wait - turning a local hiccup into a fleet-wide stall and idle (very expensive) GPUs.

What is job-to-job interference and why is it worse in AI fabrics?

Independent jobs sharing one fabric congest each other (noisy neighbour), inflating tail latency; the barrier/synchronisation effect amplifies it - one job's burst can stall another job's slowest GPU. Mitigate with tenant isolation/QoS and good load balancing.

Sources of stragglers

Congestion / incast, packet loss + retransmit (RoCEv2 is drop-sensitive, RDMA, RoCE and RoCEv2), ECMP hash collisions putting two elephant flows on one link (Load balancing and hashing - ECMP and entropy), PFC pauses / HOL blocking, a flapping optic, or host/NUMA issues.

Job-to-job interference (the multi-tenant tax)

Real clusters run many independent jobs over one fabric. Each job's synchronised collectives are already tail-latency-sensitive; when jobs share links, job-to-job interference (noisy-neighbour congestion) lengthens the tail for everyone - and the barrier/synchronisation effect amplifies it, so one job's burst can stall another job's slowest GPU. This is why isolation (QoS, tenant separation - Tenant isolation and segmentation in shared AI clusters) and good load balancing (Load balancing in AI fabrics - flow, flowlet, packet spray) matter as much as raw bandwidth in a shared cluster.

Why it matters for design

JCT is the metric, and it is dominated by the worst-case path. So AI fabrics are engineered to minimise variance: non-blocking rail-optimized topology (Rail-optimized CLOS topology), lossless transport (Lossless fabric - PFC, ECN, DCQCN), adaptive/flowlet load balancing (instead of static ECMP), reliable optics, and telemetry to locate the straggler (AI-assisted operations - AIOps). This is the why behind every other Network-section choice.

IPv6 / dual-stack note

Tail-latency sources are family-agnostic; ensure load balancing/entropy work for whichever family carries RoCEv2.

Spaced repetition

Why does tail latency (not average) dominate AI training performance?

Collectives are barrier-synchronised - every GPU waits for the slowest path/GPU (the straggler), so the worst case gates Job Completion Time.

Name three network sources of stragglers.

Congestion/incast, packet loss+retransmit (RoCEv2), ECMP hash collisions (elephant flows on one link), PFC pauses/HOL, flapping optics (any three).

The single metric AI fabric design optimises is [...], achieved by [...].

The single metric AI fabric design optimises is Job Completion Time (JCT), achieved by minimising latency variance.

Sources

  • Cisco AI/ML blueprint; Nvidia/UEC collective-communication performance studies.

domain: AI-Infra · blueprint-ref: AI-Infra 2.3.c Latency considerations · type: concept · status: complete · tags: [elective/ai-infra, ai/network, tradeoff/latency, tradeoff/convergence]

Read more