AI: Network

Parallelism strategies and collective traffic patterns

Miro Kosut

Miro Kosut

— 4 min read
Share

The bridge between the workload and the fabric: how a model is split across GPUs decides the traffic the network must carry. A network architect doesn't train the model, but must know which parallelism strategy and which collective operations produce the east-west load (AI-ML workloads and their infrastructure impact).

Why split at all

A single GPU's memory cannot hold a large model's weights, activations, gradients, and optimizer state - so computation is distributed across GPUs, which must then synchronise over the network. Intra-node exchange rides NVLink/PCIe (DMA, ~TB/s); inter-node exchange rides the RDMA backend fabric (RDMA, RoCE and RoCEv2).

The four parallelism strategies (and their traffic)

  • Data parallelism - every GPU holds a full model copy and processes different mini-batches; after the backward pass the gradients are averaged across all GPUs (sum / N). Traffic = a periodic, synchronised All-Reduce of the whole gradient set - bursty, all-GPUs-at-once, and the dominant pattern at scale.
  • Pipeline parallelism (a form of model parallelism) - consecutive layers sit on different GPUs and the batch is cut into micro-batches that flow stage to stage. Traffic = point-to-point activations between adjacent stages; the hazard is the pipeline "bubble" (GPUs idle while the pipeline fills and drains).
  • Tensor parallelism - a single layer's matrices are sharded across GPUs (e.g. Q/K/V in attention). Needs All-Gather to assemble shards in the forward pass and All-Reduce for gradients - very chatty and latency-critical, so it is normally kept inside one node over NVLink.
  • Expert parallelism (MoE) - different experts live on different GPUs and a router sends each token to its expert. Traffic = All-to-All (tokens dispersed then regathered) - the hardest pattern to balance.

Hybrid "3D" parallelism combines data + pipeline + tensor (and sometimes expert) in one job, so a real cluster carries several of these patterns at once.

Parameter server vs all-reduce

Early data-parallel training used a parameter server (workers push gradients to a central server, pull back updates) - simple, but the server is a bandwidth bottleneck and a single point of failure. Modern training uses peer All-Reduce (no central server), which is bandwidth-optimal and the reason the backend fabric is an any-to-any Clos.

Collective operations (what NCCL actually runs)

NCCL (NVIDIA Collective Communication Library) is the topology-aware library GPUs use to communicate; it builds ring or tree topologies and supports AllReduce, Broadcast, Reduce, AllGather, ReduceScatter (plus All-to-All for MoE). Two matter most:

  • Broadcast (usually a tree) - push identical starting weights to every GPU at job start.
  • All-Reduce (usually a ring) - the gradient-sync workhorse, implemented as ReduceScatter then AllGather: each GPU owns and reduces one chunk, then the reduced chunks are gathered back to all. Ring all-reduce is bandwidth-optimal - per-link volume stays roughly constant as the GPU count grows.

GPUs bootstrap via a rendezvous: a master (rank 0) distributes an NCCL Unique ID over TCP so all ranks join the same group and build the ring/tree; inter-host pairs then talk over RDMA Queue Pairs.

Why it matters for design

The parallelism mix sets the traffic matrix: data-parallel = periodic synchronised all-reduce (huge elephant flows -> Load balancing in AI fabrics - flow, flowlet, packet spray); tensor = NVLink-bound chatter (keep intra-node); pipeline = stage-to-stage (sensitive to the slowest hop); MoE = all-to-all (worst to balance). All of it is barrier-synchronised, so the slowest GPU/link gates the whole job (Latency and the straggler problem) and any loss stalls training (Lossless fabric - PFC, ECN, DCQCN). This is why the fabric is non-blocking, lossless, and any-to-any (Validated reference design - Cisco AI-ML lossless fabric (CVD)).

What would change this (mid-scenario twist)

  • Model fits one GPU -> pure data parallelism (only all-reduce to plan for).
  • Model too big -> tensor (intra-node) + pipeline (inter-node) + data (across replicas): plan NVLink and a strong backend.
  • Mixture-of-Experts -> budget for All-to-All; flow-based ECMP will hot-spot badly.

IPv6 / dual-stack note

Collective/RDMA endpoints are addressed like any host; run the backend dual-stack and ensure the chosen load-balancing (hashing/spray) is consistent across v4/v6.

Spaced repetition

The four parallelism strategies and their dominant traffic?

Data (All-Reduce of gradients), pipeline (stage-to-stage activations + bubble), tensor (intra-layer All-Gather + All-Reduce, NVLink-bound), expert/MoE (All-to-All). Hybrid "3D" combines them.

Why is tensor parallelism normally kept inside a single node?

It shards one layer's matrices and needs heavy, latency-critical All-Gather/All-Reduce every step - volumes only NVLink/PCIe can carry without hurting step time.

All-Reduce over a ring is implemented as which two phases, and why use it?

ReduceScatter then AllGather; ring all-reduce is bandwidth-optimal (per-link traffic roughly independent of GPU count), which is why it replaced the parameter-server model.

What does a parameter server cost you versus peer all-reduce?

The central server is a bandwidth bottleneck and single point of failure; peer all-reduce removes it and is bandwidth-optimal, needing an any-to-any fabric.

What bootstraps NCCL collective communication across nodes?

A rendezvous - the rank-0 master distributes an NCCL Unique ID over TCP so all ranks join one group and build the ring/tree; inter-host GPUs then use RDMA Queue Pairs.

Sources

  • Toni Pasanen, Deep Learning for Network Engineers (2025), Ch 8 (Parallelism Strategies) and Ch 14 (GPU Cluster Communication Model); NVIDIA NCCL.

domain: AI Infrastructure · type: design-note · status: complete · tags: · source: Pasanen, "Deep Learning for Network Engineers" (2025), Ch 8 & 14

Read more