Core: Data Plane Transport

Buffering, microbursts, and congestion signaling

Miro Kosut

Miro Kosut

— 4 min read
Share

Requirement / business driver

Absorb transient bursts and signal congestion without either dropping latency-sensitive traffic or adding bufferbloat - a core data-centre and WAN-edge decision.

The forces

  • Microbursts: sub-second spikes that overrun interface buffers even when average utilisation is low (see Factors affecting convergence) - they cause drops that average graphs never show.
  • Incast: many senders -> one receiver (storage reads, AI all-reduce) momentarily swamp a port's buffer.
  • Deep vs shallow buffers: deep-buffer switches ride out bursts/incast but can add latency/bufferbloat; shallow-buffer cut-through switches give low latency but drop sooner. The choice is a DC design decision tied to workload.
  • Signaling: tail-drop is blunt; AQM/WRED drops early to keep queues short; ECN marks instead of dropping (and DCQCN uses ECN to drive RoCE flow control in lossless fabrics - Lossless fabric - PFC, ECN, DCQCN).

DC transport has a TCP-side refinement of the same idea: DCTCP uses ECN not as a binary signal but as a measure - switches mark on instantaneous queue length, and the sender scales its window back in proportion to the fraction of marked packets - producing smooth, low-variance sending rates and short queues without lossless machinery; it is the TCP cousin of DCQCN (which applies proportional ECN response to RoCE), and the DC answer when east-west TCP latency matters more than raw throughput (TCP behavior and the network).

Where the buffer sits - placement architectures

Orthogonal to deep-vs-shallow is buffer placement. A naive design - one shared ingress buffer feeding a crossbar - suffers head-of-line blocking: when one egress congests, its backpressure stalls frames queued behind the blocker even though their own egress ports are idle. Virtual output queuing (VOQ) removes this by giving every ingress a separate queue per egress port and class (an NxN queue structure) and letting a scheduler grant fabric access only when the egress can accept - congestion is buffered at the ingress, drops are confined there, and no destination can block another's path. VOQ platforms typically pair the on-chip buffer with deep external memory (HBM), which is what lets modular chassis ride out severe incast and large speed mismatches (a 400G fabric feeding lower-rate edge or WAN links). The design dominant in fixed leaf/ToR silicon is the opposite: a shared-memory egress (output-queued) pool - every port on the ASIC (or slice) draws from one egress buffer sized N rather than NxN, each queue's cap is computed dynamically as a weight times the currently free memory so no congested queue can monopolise the pool, and flow-aware policies (AFD/DPP - Lossless fabric - PFC, ECN, DCQCN) keep headroom free for bursts. Placement follows the failure mode: deep-buffer VOQ boxes earn their cost at speed-mismatch, border, and storage-facing roles; shared-egress shallow switches win at latency-sensitive leaf roles where ECN keeps queues short - and on lossless (PFC) classes a bigger buffer is not automatically better, since PFC headroom and pause chains scale with it. Load balancing in AI fabrics - flow, flowlet, packet spray shows the same VOQ idea lifted to fabric scale (Scheduled Ethernet).

Recommendation / justification

Match buffers to workload: deep buffers where incast/bursty bulk dominates, shallow/low-latency where tail latency rules. Prefer ECN/AQM over tail-drop so TCP backs off without costly retransmits (TCP behavior and the network); reserve priority queuing for real-time (QoS design - models and strategy).

What would change this (mid-scenario twist)

  • Storage/AI incast appears -> deeper buffers and/or ECN-based control (lossless for RoCE).
  • Latency SLA tightens -> shallow buffers + strict priority + ECN.

Validation checks

  • Are microburst drops visible (microburst/queue telemetry, not 5-min averages)?
  • Is congestion signalled (ECN/WRED) rather than only tail-dropped?

IPv6 / dual-stack note

Buffering/ECN are L3/L4-agnostic; ECN bits live in the IPv6 Traffic Class field too.

Spaced repetition

Why can a link with low average utilisation still drop packets?

Microbursts - sub-second spikes overrun the buffer between polling intervals, invisible to averaged graphs.

Deep-buffer vs shallow-buffer switches trade [...] against [...].

Deep-buffer vs shallow-buffer switches trade burst/incast absorption (but added latency/bufferbloat) against low latency (but earlier drops).

ECN is preferable to tail-drop because [...]; in RoCE fabrics ECN drives [...].

ECN is preferable to tail-drop because it signals congestion by marking, so TCP slows without retransmitting; in RoCE fabrics ECN drives DCQCN.

What problem does virtual output queuing (VOQ) solve inside a switch, and how?

Head-of-line blocking. With one shared ingress buffer, a congested egress backpressures traffic bound for every other egress; VOQ gives each ingress a queue per egress port and class (an NxN structure) and a scheduler grants fabric access, so congestion is buffered at ingress and drops are confined there.

In a shared-memory egress switch, a queue's cap is computed [...], so one congested queue cannot monopolise the shared pool.

In a shared-memory egress switch, a queue's cap is computed dynamically as a weight times the currently free buffer, so one congested queue cannot monopolise the shared pool.

[...] reacts in proportion to the fraction of ECN-marked packets (marked on instantaneous queue depth) - smooth low-latency DC TCP, the TCP-side cousin of DCQCN.

DCTCP reacts in proportion to the fraction of ECN-marked packets (marked on instantaneous queue depth) - smooth low-latency DC TCP, the TCP-side cousin of DCQCN.

Sources

  • Cisco Press, End-to-End QoS Network Design, 2nd Ed. (Szigeti et al.) - DCTCP.
  • Cisco Press, End-to-End QoS; Cisco/Nvidia AI/ML fabric guidance (incast, ECN).
  • Cisco Live BRKDCN-3953, End-to-End QoS Implementation and Operation with Nexus (buffer placement architectures).

domain: Core · blueprint-ref: Core 4.2 Characteristics and properties (buffering/congestion) · type: design-decision · status: complete · tags: [core, core/dataplane, tradeoff/latency, tradeoff/convergence]

Read more