Fabric transport - Ethernet (RoCEv2) vs InfiniBand vs UEC
The signature AI-Infra design decision: what transport carries the backend / scale-out GPU fabric (the east-west, RDMA, highest-speed network that GPUs use for collective communication). Get this wrong and every other choice inherits the pain.
Requirement / business driver
Move synchronized collective traffic (all-reduce / reduce-scatter) between GPUs without becoming the bottleneck on Job Completion Time (JCT). Training is barrier-synchronized, so the fabric must deliver consistent low latency and effectively zero loss while fitting the customer's budget, operating model, and scale.
Constraints
- Technical: lossless or loss-tolerant transport; line-rate bursts from many GPUs at once; tail latency matters more than average.
- Operational: does the customer's team already run Ethernet, or do they have an HPC/InfiniBand skill set?
- Application: training (bandwidth-hungry, loss-sensitive) vs inference (latency/availability, lighter east-west).
- Business / cost: hardware cost, and the cost of running a second operational stack.
- Regulatory / sovereignty: usually transport-neutral, but air-gap/sovereignty pushes toward on-prem and influences vendor choice.
- Dual-stack (IPv4 + IPv6): the backend fabric is typically a private, routed underlay - design it dual-stack like any L3 fabric (see IPv6 note).
Options
- A - Ethernet / RoCEv2. RDMA over routable UDP/IP; losslessness engineered with PFC + ECN + DCQCN. Open, multi-vendor, converged.
- B - InfiniBand. Purpose-built HPC fabric; lossless by design (credit-based flow control); lowest, most deterministic latency; single-vendor (NVIDIA).
- C - Ultra Ethernet (UEC). Open Ethernet transport (UET) purpose-built for AI - packet spraying, modern congestion control, low tail latency; spec 1.0 released June 2025, products emerging.
Comparison
Tradeoffs
- Ethernet / RoCEv2 wins when the customer wants an open, multi-vendor fabric, **reuse
of existing Ethernet operations, a converged* fabric (compute + storage), and lower hardware cost. Watch-outs: you must engineer* losslessness - PFC/ECN/DCQCN thresholds, buffer headroom, and PFC-storm/deadlock avoidance (see Lossless fabric - PFC, ECN, DCQCN).
- InfiniBand wins for a turnkey, maximum-performance cluster where the customer accepts
single-vendor lock-in (the classic NVIDIA DGX SuperPOD pattern) and has (or will build) the separate skill set. Watch-outs: lock-in, separate tooling, separate compute vs storage networks, higher cost, subnet-scale ceiling.
- UEC is the strategic open path at very large scale - it ports InfiniBand-class ideas
onto open Ethernet and targets millions of endpoints. Watch-outs: it is new - confirm the customer's chosen vendors actually ship conformant UET silicon/NICs before you commit a design to it.
One more axis separates the poles: who runs the fabric. InfiniBand is centrally managed - a Subnet Manager computes and programs forwarding for the whole subnet, and a UFM-class fabric manager delivers near zero-configuration operation with self-healing rerouting - while RoCEv2 is fully distributed: a standard IP control plane, device-by-device configuration, which is exactly why the PFC/ECN/DCQCN tuning burden lands on your operators. Centralised turnkey versus distributed-but-yours is as real a fork dimension as latency or cost - it decides whose expertise the fabric depends on.
Recommendation
For the typical enterprise, default to Ethernet / RoCEv2 today: it satisfies the open-ecosystem, operational-reuse, and cost requirements while meeting JCT with correct lossless design. Reserve InfiniBand for turnkey max-performance clusters where lock-in is acceptable, and treat UEC as the forward-looking choice you adopt as it matures (especially greenfield, very large scale).
Justification
The decision is requirement-driven, not latency-driven. At modern speeds NCCL collective bandwidth is near-parity between a well-built RoCEv2 fabric and InfiniBand (large hyperscaler training clusters have been built successfully on both), so the deciding axes become ecosystem, operability, cost, and scale - and on those, open Ethernet aligns with most enterprise requirements. You justify InfiniBand only when the requirement explicitly prizes turnkey peak performance over openness/cost.
Decision tree
Ultra Ethernet specifics, and migrating off InfiniBand
Two practical points the option above understates. First, what UET actually changes: the Ultra Ethernet transport (UET) moves RDMA reliability and congestion control into a modern multipath stack, so it does not require a strictly lossless network the way RoCEv2 does. It load-balances with multipath packet spraying (not ECMP-pinned flows), lets the application express flexible packet ordering, runs coordinated sender + receiver congestion control across paths, and on congestion can truncate a packet to deliver its header plus congestion state and recover with selective acknowledgement instead of RoCE's heavyweight Go-Back-N. An optional Link Level Reliability (LLR) extension adds fast hardware failover on a degrading link, and it plugs into existing AI/HPC frameworks through Libfabric as the northbound API, so workloads migrate without code changes. Net design effect: UET relaxes the strict PFC/ECN engineering RoCEv2 demands while keeping Ethernet's ecosystem - which is why it is the strategic open path at very large scale.
Second, migration is phased, not rip-and-replace. RoCEv2 protects the existing Ethernet investment: you upgrade host NICs gradually and stand up lossless islands (PFC/ECN enabled) that grow as adoption spreads, while socket-based legacy apps keep working over the same fabric and native RDMA apps use the verbs API to bypass the kernel for full performance. A real InfiniBand-to-Ethernet cutover is still planned in advance in phases (assess workloads -> build the lossless island -> move nodes -> validate), not flipped in one step.
What would change this (mid-scenario twist)
- Existing InfiniBand estate / HPC team appears -> lean InfiniBand (reuse skills + gear).
- Cost becomes the dominant constraint -> lean Ethernet.
- Data sovereignty / air-gap is mandated -> on-prem; the choice stays transport-neutral but design strict isolation.
- Cluster is small (<= a few hundred GPUs) -> a rail-only Ethernet design is often simplest (see Rail-optimized CLOS topology).
- "No vendor lock-in" is a hard requirement -> Ethernet now, UEC as it matures.
Validation checks
- Can the design absorb a synchronized all-reduce burst with no drops on the lossless queue (buffer/headroom math consistent with PFC/ECN thresholds)?
- Is there no cyclic buffer dependency that could deadlock PFC, and is a PFC watchdog in place?
- Is the backend fabric non-blocking (1:1) where collective traffic demands it?
- Does the operating model match the team's skills (don't hand an Ethernet team an InfiniBand fabric)?
IPv6 / dual-stack note
The backend GPU fabric is a private routed underlay; the exam is dual-stack throughout, so be ready to address it with IPv6 (or dual-stack) and to discuss v6 implications for the underlay, management, and any storage-over-IP paths. RoCEv2 rides UDP/IP and is agnostic to v4/v6.
Related
- Lossless fabric - PFC, ECN, DCQCN
- RDMA, RoCE and RoCEv2
- Rail-optimized CLOS topology
- Scale-up vs scale-out
- Optics and cabling selection
- Latency and the straggler problem
Spaced repetition
Why does packet loss hurt an AI training fabric so much more than a typical enterprise fabric?
Training is barrier-synchronized (collective ops), so the whole job waits on the slowest flow - loss/tail latency directly extends Job Completion Time (the straggler problem).
RoCEv2 achieves losslessness on Ethernet using the combination of [...].
RoCEv2 achieves losslessness on Ethernet using the combination of PFC + ECN (DCQCN).
InfiniBand is lossless by design because it uses [...] built into the protocol.
InfiniBand is lossless by design because it uses credit-based flow control built into the protocol.
The single biggest argument against InfiniBand in an enterprise design is [...].
The single biggest argument against InfiniBand in an enterprise design is vendor lock-in (single-vendor / NVIDIA).
UET (the Ultra Ethernet transport) reduces tail latency partly by using [...] across the fabric instead of pinning a flow to one path.
UET (the Ultra Ethernet transport) reduces tail latency partly by using packet spraying across the fabric instead of pinning a flow to one path.
At modern speeds the Ethernet-vs-InfiniBand choice is decided mainly by ecosystem, operability, cost and scale - not by [...] (collective bandwidth is near-parity).
At modern speeds the Ethernet-vs-InfiniBand choice is decided mainly by ecosystem, operability, cost and scale - not by latency (collective bandwidth is near-parity).
How does Ultra Ethernet Transport (UET) change the lossless requirement versus RoCEv2?
UET puts modern multipath transport, coordinated congestion control, and out-of-order/selective-ack recovery in the stack, so it does not need a strictly lossless (PFC-engineered) network the way RoCEv2 does; it also sprays packets instead of using ECMP and exposes Libfabric as its API.
How do you migrate InfiniBand or legacy hosts onto a RoCEv2 Ethernet fabric without a rip-and-replace?
Protect the investment with RoCE - upgrade host NICs gradually and grow lossless (PFC/ECN) islands; socket apps keep working while native RDMA apps use the verbs API; plan the cutover in phases (assess -> build island -> move nodes -> validate).
The fabric fork has a control-model axis: InfiniBand is [...] while RoCEv2 is fully distributed - the tuning burden lands on your operators.
The fabric fork has a control-model axis: InfiniBand is centrally managed (Subnet Manager + UFM-class zero-config, self-healing) while RoCEv2 is fully distributed - the tuning burden lands on your operators.
Sources
- NADDOD, Introduction to RoCE v2 Network (blog, Sep 2023; user-supplied PDF, full pass 2026-07-26) - Subnet-Manager/UFM centralised model vs distributed RoCEv2 configuration.
- Cisco, Data Center Networking Blueprint for AI/ML Applications.
- Cisco Live BRKDCN-2921, Network Best Practices for AI/ML Data Center.
- Ultra Ethernet Consortium, UEC Specification 1.0 (June 2025) overview.
- Cisco U., "InfiniBand-to-Ethernet Transition" (High-Throughput Converged Fabrics) - UET features, Silicon One, phased migration.
- NVIDIA networking docs (InfiniBand / Spectrum-X / NCCL).
- Meta engineering, 24k-GPU H100 cluster write-ups (RoCE and InfiniBand builds).