Core: Data Plane Transport

Load balancing and hashing - ECMP and entropy

Miro Kosut

Miro Kosut

— 2 min read
Share

Requirement / business driver

Spread traffic across parallel paths/links evenly and without reordering - the basis of CLOS/spine-leaf and LAG scaling.

How it works

  • ECMP / LAG hash a flow (typically the 5-tuple) to one path, keeping a flow's packets in order. Per-packet balancing spreads better but reorders (bad for TCP) - usually avoided.
  • Entropy is the variety in the hash input. Problems arise when headers are hidden: tunnels (GRE/IPsec/VXLAN) bury the inner 5-tuple, so devices balance on the outer header and may polarise. Fixes: entropy from UDP source-port (VXLAN), an MPLS entropy label (MPLS fundamentals), or the IPv6 Flow Label.
  • Elephant flows / single big flows (QUIC/UDP, large TCP transfers, storage) can't be split - one flow rides one link and can congest it while others sit idle.
  • Polarisation: identical hashing at successive tiers sends all traffic the same way - vary the hash seed per tier.

Recommendation / justification

Use flow-based ECMP (avoid per-packet reordering); maximise entropy (expose UDP source-port / MPLS entropy label / v6 flow label) and vary hash inputs per tier to prevent polarisation. For elephant-flow environments (AI all-reduce, storage) recognise that static hashing leaves links hot/cold - hence adaptive/flowlet routing in AI fabrics (Fabric transport - Ethernet vs InfiniBand vs UEC).

What would change this (mid-scenario twist)

  • Tunnelled traffic polarises -> enable entropy label / source-port entropy.
  • A few elephant flows dominate -> adaptive/flowlet load balancing, not plain ECMP.

Validation checks

  • Is balancing flow-based (no reordering) with enough entropy through tunnels?
  • Are hash inputs varied per tier to avoid polarisation?

IPv6 / dual-stack note

The IPv6 Flow Label is a first-class entropy source - have transit devices include it in the hash for encapsulated/IPv6 flows.

Spaced repetition

Why is flow-based ECMP preferred over per-packet load balancing?

Per-packet spreads more evenly but reorders packets, hurting TCP; flow-based keeps a flow on one path in order.

Tunnels (GRE/IPsec/VXLAN) hurt load balancing because [...]; fixes include [...].

Tunnels (GRE/IPsec/VXLAN) hurt load balancing because they hide the inner 5-tuple, lowering entropy and causing polarisation; fixes include UDP source-port entropy, MPLS entropy label, or IPv6 flow label.

A single elephant flow is a problem for ECMP because [...] (hence adaptive/flowlet routing in AI fabrics).

A single elephant flow is a problem for ECMP because one flow maps to one link - it can't be split, so that link congests while others idle (hence adaptive/flowlet routing in AI fabrics).

Sources

  • Cisco Press, Optimal Routing Design; CLOS / data-centre fabric guidance.

domain: Core · blueprint-ref: Core 4.2 Characteristics (load balancing/hashing) · type: design-decision · status: complete · tags: [core, core/dataplane, tradeoff/scale, tradeoff/convergence]

Read more