AI: Network

Lossless fabric - PFC, ECN, DCQCN

Miro Kosut

Miro Kosut

— 9 min read
Share

In one line

The set of mechanisms that make an Ethernet fabric near-lossless so RoCEv2 RDMA can run without the retransmits that would wreck Job Completion Time.

How it works

  • ECN (Explicit Congestion Notification) - end-to-end, early. A congested switch marks the

    IP ECN bits; the receiver returns a Congestion Notification Packet (CNP); the sender slows that flow. On Cisco Nexus the marking is done by WRED / Approximate Fair Drop (AFD).

  • PFC (Priority Flow Control) - hop-by-hop, last resort. The switch sends a per-priority

    PAUSE to the upstream device to stop drops. For routed RoCEv2 the priority is keyed on DSCP, and PFC frames use link-local addressing so they work across routed and switched hops; the no-drop queue raises pause at its xOFF threshold and releases below xON.

  • DCQCN = ECN + PFC working together: ECN does the gentle, early, end-to-end rate reduction

    so that PFC (which is blunt and can pause traffic it didn't mean to) fires as rarely as possible.

Why it matters for design

A single dropped packet forces an expensive retransmit, and because training is barrier-synchronized one slow flow stalls the whole job (the straggler problem). So the fabric is engineered for zero loss on the RDMA queue, not merely low average latency. This is the price of admission for Ethernet vs InfiniBand.

Design knobs & hazards

  • Buffer headroom + ECN K-min / K-max thresholds must be mathematically consistent with the

    buffer allocated to the lossless queue. Misaligned DCQCN thresholds are the most common RoCEv2 problem.

  • PFC hazards: deadlock (cyclic buffer dependency), PFC storms / pause propagation, and

    head-of-line blocking. Mitigate with careful buffer design, avoiding cyclic dependencies, and a PFC watchdog that detects stuck/persistent pause and acts - and its action is worth naming: it clears the stuck no-drop queue, a deliberate drop inside the lossless class. The watchdog trades the lossless guarantee for fabric liveness, cutting the pause chain before a single malfunctioning NIC or port freezes every upstream sender of that class (the fabric-wide freeze documented in Microsoft's RDMA-at-scale experience). Losslessness in production is therefore always bounded by a liveness escape hatch - a design admission, not a bug.

  • Cisco buffer features to name: Dynamic Buffer Protection (DBP) for fair shared-buffer use;

    AFD + ETRAP (elephant trap) + Dynamic Packet Prioritization (DPP) to separate mice from elephant flows (*WRED marks every packet in the queue equally; AFD marks only the high-rate elephant flows, proportional to their rate, leaving mice unmarked so short flows run to completion*); dedicated per-port buffering on the GPU backend improved collective performance in Cisco testing.

  • Contrast: InfiniBand needs none of this - it is lossless by design via credit-based flow control.

The DCB suite (what "lossless Ethernet" is built from)

"Lossless Ethernet" is really the Data Center Bridging (DCB) family working together: PFC (per-priority pause, the no-drop guarantee), ETS (Enhanced Transmission Selection - allocates a guaranteed share of bandwidth per traffic class so the lossless class and others coexist; scheduled with DWRR), and DCBX (DCB Exchange - lets switch and NIC auto-negotiate the PFC/ETS parameters so both ends agree). PFC provides the no-drop behaviour; ETS keeps it from starving everything else; DCBX keeps host and fabric consistent.

Validated reference values (Cisco CVD)

One tested reference point for the abstract knobs above (Cisco's AI/ML CVD, NX-OS on Nexus 9000):

  • Marking: hosts mark RoCEv2 with DSCP 24 (CS3) -> a dedicated no-drop queue given roughly 60% of the bandwidth, with WRED + ECN on that queue (reference thresholds about 150 KB min / 3000 KB max, ~7% drop probability, tuned for 100G/400G ports).
  • Protect the feedback: CNP is marked DSCP 48 and placed in a strict-priority queue, so congestion notifications are themselves never dropped or delayed.
  • Headroom via MTU: the fabric runs jumbo MTU (9216); on the no-drop class the MTU is what the switch uses to size PFC headroom (it does not change forwarding).
  • Where it attaches: network-qos (PFC) and queuing/ECN are applied system-wide, while classification and PFC-on are per-interface - and the PFC watchdog (about 100 ms) is on every port.
  • Ordering & participation: tune so *ECN marks before PFC pauses (ECN is the end-to-end first line, PFC the hop-by-hop backstop), and remember endpoints must run ECN + PFC* - the network alone cannot make the fabric lossless. A full worked design: Validated reference design - Cisco AI-ML lossless fabric (CVD).

Inside DCQCN: the sender's rate machine

DCQCN (Microsoft, SIGCOMM 2015) fuses QCN's rate control with DCTCP's ECN marking, and names three roles: the Congestion Point (the switch that marks), the Notification Point (the receiving NIC that returns CNPs), and the Reaction Point (the sending NIC that adjusts). Unlike TCP's in-flight window, RoCE congestion control is rate-based, run entirely in NIC hardware on a short per-flow (QP) control loop - roughly every 45 microseconds by default. The decrease side keeps a moving average of congestion probability from the fraction of marked traffic and cuts the current rate in proportion - one marked packet trims it barely, everything marked halves it - while the pre-cut rate is remembered as the target. The increase side then climbs in two phases: Fast Recovery (a few periods halving the distance back to the remembered target - what keeps steady-state utilisation high), then Active Increase (nudging the target up by a fixed increment each period to probe for spare bandwidth). Two design forks sit on top. Marking policy: DCQCN-p marks probabilistically along a WRED ramp - few CNPs, a bare congestion-present signal, and with many competing flows the queue stabilises deeper; DCQCN-d marks deterministically (everything above one threshold, DCTCP-style) - the CNP stream becomes a multibit measure of congestion severity, at the price of far heavier CNP processing on the NIC. Vendor variance: the algorithm lives in the NIC and vendors ship different refinements - jitter on the control loop to desynchronise flows (avoiding lockstep burst-and-silence oscillation), incast-aware pacing that slows probing when CNP timing implies a wide incast, superlinear post-congestion ramps in place of the original hyper-increase. Consequences for design: DCQCN behaviour is a NIC property as much as a switch property, so a mixed-NIC estate runs mixed congestion control; and the cascade analysis exposes the trap the thresholds must avoid - once xOFF engages upstream, paused traffic stops arriving marked, CNPs dry up as their timers expire, and every Reaction Point's increase phase re-accelerates into a fabric that is choking, relocating pressure hop-by-hop toward the hosts. PFC does not merely pause traffic; it silences the ECN control loop - the sharpest reason WRED must act early enough that pauses stay rare, brief, and local.

InfiniBand's side of the fork runs the same feedback loop under different names: FECN/BECN - a congested switch sets the forward notification toward the receiver, which reflects a backward notification to the sender, with marking rates adapted to congestion depth. Like ECN it is coarse-grained - a mark says slow down, not by how much - which is why neither family's marking, by itself, outperforms the other; the differentiation lives in what the endpoints do with the signal.

A production counter-doctrine: PFC-only with receiver-driven admission

DCQCN is the textbook answer, not a law of nature. Meta's production RoCEv2 training clusters dropped it at the 400G generation: retuning stumbled on NIC-firmware regressions (mis-counted CNPs, lost visibility), and the fabric has since run for years with PFC as the only network congestion control - stably. What replaces the transport loop is a pair of structural substitutes. First, receiver-driven traffic admission, co-designed into the collective library: a sender's proxy may post an RDMA write only after the receiver returns a clear-to-send (CTS) grant for a ready channel buffer, so channel count x channel-buffer size becomes the in-flight window that caps congestion at the source. Sizing that window is the new tuning problem - too large spreads congestion, too small under-fills the pipe (RoCE's flow control is coarser than InfiniBand's credits) - and the CTS grants ride a high-priority class so the feedback loop cannot be starved by the data it governs. Second, buffer asymmetry in the topology: a deep-buffered spine absorbs the collective bursts (Rail-optimized CLOS topology), and in years of production the spine has never had to push persistent PFC back toward the leaves. The design lesson: the congestion-control function can migrate out of the transport into the collective library plus buffer architecture, at the price of tight coupling between library, NIC, and network - a posture to weigh against DCQCN, not a general-purpose replacement (NVIDIA's guidance agrees at the mild end: PFC-only or ECN-only deployments are viable with capable NICs, coupling both is best). Two field notes travel with this. A PFC pause frame carries a per-CoS timer in pause quanta (one quantum = 512 bit-times; zero quanta = resume), which is what makes the pause per-priority and self-expiring. And the RoCEv2 DSCP value is a convention, not a standard - Cisco's CVD marks data CS3/24 while NVIDIA examples use AF31/26 - so what matters is end-to-end agreement between NIC and every switch, which is exactly what DCBX exists to negotiate.

Spaced repetition

In DCQCN, what is the role of ECN vs PFC?

ECN gives early, end-to-end, per-flow rate reduction; PFC is the blunt hop-by-hop last resort. ECN minimizes how often PFC has to fire.

The mechanism that detects a stuck/persistent PFC pause and takes corrective action is the [...].

The mechanism that detects a stuck/persistent PFC pause and takes corrective action is the PFC watchdog.

The most common cause of RoCEv2 problems in practice is [...].

The most common cause of RoCEv2 problems in practice is misaligned DCQCN / ECN thresholds (vs buffer headroom).

A cyclic buffer dependency under PFC can cause [...].

A cyclic buffer dependency under PFC can cause deadlock.

In the Cisco AI/ML CVD, how are RoCEv2 data and CNP marked and queued?

RoCEv2 = DSCP 24 (CS3) into a no-drop queue (~60% bandwidth, WRED+ECN); CNP = DSCP 48 into a strict-priority queue so congestion feedback is never dropped.

WRED vs AFD for ECN marking?

WRED marks every packet in the no-drop queue equally; AFD is flow-aware - it marks only high-rate elephant flows (proportional to rate) and leaves mice unmarked, so short flows finish and only the biggest congestors slow down.

What three DCB components make up "lossless Ethernet," and their roles?

PFC (per-priority no-drop pause), ETS (Enhanced Transmission Selection - guaranteed per-class bandwidth, DWRR-scheduled), and DCBX (negotiates PFC/ETS parameters between switch and NIC).

What does the PFC watchdog actually do when it fires, and what does that trade?

It clears (drops) the stuck no-drop queue - a deliberate drop inside the lossless class - trading the lossless guarantee for fabric liveness so one malfunctioning NIC/port cannot freeze every upstream sender; production losslessness is always bounded by this escape hatch.

What did Meta run instead of DCQCN on its 400G RoCE training fabrics, and why does it work?

PFC as the only network congestion control, plus receiver-driven admission in the collective library - a sender posts an RDMA write only after a clear-to-send (CTS) grant, so channels x channel-buffer size cap in-flight traffic - with a deep-buffered spine absorbing bursts. CC moved from the transport into the library + buffer architecture.

In receiver-driven traffic admission, the grant a sender must receive before posting an RDMA write is the [...] packet - expedited in a high-priority class so the feedback loop is never starved.

In receiver-driven traffic admission, the grant a sender must receive before posting an RDMA write is the clear-to-send (CTS) packet - expedited in a high-priority class so the feedback loop is never starved.

DCQCN's two-phase rate increase after a CNP-driven cut?

Fast Recovery - a few control periods halving the distance back to the remembered pre-cut target (keeps steady-state utilisation high) - then Active Increase, nudging the target up by a fixed increment per period to probe for spare bandwidth. Rate-based, per-QP, in NIC hardware on a ~45-microsecond loop.

Why is a PFC pause worse than "just waiting" for DCQCN?

Paused traffic stops arriving ECN-marked, CNPs dry up as timers expire, and senders' increase phases re-accelerate into a choking fabric - PFC silences the ECN control loop and pressure relocates upstream hop-by-hop, so WRED must fire early enough that pauses stay rare and local.

[...] marks probabilistically along a WRED ramp (few CNPs, a congestion-present signal, deeper queues at scale); [...] marks deterministically above one threshold (the CNP stream becomes a multibit severity measure, heavier NIC load).

DCQCN-p marks probabilistically along a WRED ramp (few CNPs, a congestion-present signal, deeper queues at scale); DCQCN-d marks deterministically above one threshold (the CNP stream becomes a multibit severity measure, heavier NIC load).

Sources

  • NADDOD, Introduction to RoCE v2 Network (blog, Sep 2023; user-supplied PDF, full pass 2026-07-26) - FECN/BECN mechanics and the coarse-grained-marking symmetry.
  • WWT, Understanding DCQCN (M. Witte, 2024) - WRED/xOFF cascade walk-through incl. the CNP-silencing dynamic; full-text pass 2026-07-25.
  • Broadcom, Introduction to Congestion Control for RoCE (NCC-WP103, M. Voloshin, 2023) - rate machine (Fast Recovery/Active Increase, CP decrease), DCQCN-p/-d, NIC-side enhancements; full-text pass 2026-07-25.
  • Meta Engineering (Gangidi and Zeng), RoCE Networks for Distributed AI Training at Scale (2024) - DCQCN-off posture, receiver-driven admission, deep-buffer spine; full-text pass 2026-07-25.
  • NVIDIA/Mellanox and Cisco, Benefits of RDMA over Routed Fabrics white paper - PFC-only / ECN-only deployment modes, CoS+DSCP marking practice; full-text pass 2026-07-25.
  • Cisco, Data Center Networking Blueprint for AI/ML Applications.
  • Cisco Live BRKDCN-2921 (AI/ML network best practices) - PFC storm/watchdog behavior; deck-level pass 2026-07-04.
  • NVIDIA, RDMA over Converged Ethernet (RoCE) docs.
  • Vendor DCQCN configuration guides (Cisco / Juniper).

domain: AI-Infra · blueprint-ref: AI-Infra 2.1 Lossless fabrics / 2.2 QoS in lossless fabrics · type: concept · status: complete · tags:

Read more