AI: Network

RDMA, RoCE and RoCEv2

Miro Kosut

Miro Kosut

— 3 min read
Share

In one line

RDMA lets a NIC read/write remote memory directly - bypassing the CPU and kernel - which is what makes GPU-to-GPU communication fast enough for training; RoCE carries RDMA over Ethernet.

RDMA

Zero-copy, kernel-bypass, NIC-offloaded transfers via the verbs API: microsecond latency, line-rate throughput, near-zero CPU. GPUDirect RDMA lets the NIC move data straight to/from GPU memory, so collective libraries (NCCL) move tensors between GPUs across nodes without touching host CPU.

Transports for RDMA

  • InfiniBand - native RDMA fabric.
  • iWARP - RDMA over TCP (less common for AI).
  • RoCE - RDMA over Ethernet. RoCEv1 is L2 (Ethertype); RoCEv2 runs over UDP/4791 and is routable over L3 - this is the one used in modern AI fabrics.

RDMA operations: memory vs channel semantics

RDMA is message-based (not TCP's byte stream), driven through the verbs API over a queue pair - a send queue + receive queue, with a completion queue signalling done. There are two operation styles:

  • Channel semantics (Message Verbs) - SEND / RECEIVE: the receiver posts a buffer and its CPU is involved; reliable or unreliable; good for short control messages.
  • Memory semantics (Memory Verbs) - READ / WRITE (plus atomics): one side reads/writes the other's registered memory without involving the remote CPU; reliable, asynchronous; built for bulk transfers.

AI/ML uses memory-semantic READ/WRITE for GPU-to-GPU exchange - the remote CPU stays out of the path, which is what makes microsecond, line-rate collectives possible. It is also why packet-spray load balancers favour reorder-tolerant WRITE-only traffic (Load balancing in AI fabrics - flow, flowlet, packet spray).

Why it matters for design

RoCEv2 is drop-sensitive: a lost packet stalls the transfer (go-back-N), so the fabric must be lossless or near-lossless via PFC/ECN/DCQCN (Lossless fabric - PFC, ECN, DCQCN). The transport choice (RoCEv2 vs InfiniBand vs UEC) is the fabric decision in Fabric transport - Ethernet vs InfiniBand vs UEC; the same drop-sensitivity is why hashing/entropy (Load balancing and hashing - ECMP and entropy) and topology matter so much.

Loss handling and the observability blind spot

Two practical corollaries of the transport design. First, "lossless required" is a performance requirement, not a protocol absolute: RoCEv2 does recover from a drop - the receiver returns a NACK carrying the missing packet sequence number (PSN) and the sender resends - but recovery is go-back-N, so on a lossy fabric the retransmit stalls erase RDMA's advantage; you can run RoCE lossy, you just forfeit the point of running it. Second, kernel bypass blinds host capture: RDMA traffic never traverses the host network stack, so tcpdump or Wireshark on the server sees nothing unless the NIC is explicitly set to mirror it. Verifying RoCE behaviour therefore has to be designed in - NIC counters and capture flags at the edge, and fabric-side flow telemetry with ECN-mark, per-CoS pause, and CNP counters in the network (AI-assisted operations - AIOps).

IPv6 / dual-stack note

RoCEv2 over UDP works for v4 and v6; backend fabrics are commonly v4 but v6 is viable - keep MTU/headroom consistent either way.

Spaced repetition

What does RDMA provide, and why does AI need it?

Zero-copy, kernel-bypass, NIC-offloaded remote-memory access (microsecond latency, low CPU) - GPUDirect RDMA moves data straight between GPU memories across nodes for collectives.

RoCEv1 vs RoCEv2: v1 = [...]; v2 = [...] (the one used).

RoCEv1 vs RoCEv2: v1 = L2 (Ethertype, non-routable); v2 = RDMA over UDP/4791, routable over L3 (the one used).

Why does RoCEv2 demand a lossless fabric?

It is drop-sensitive - a lost packet stalls the transfer - so PFC/ECN/DCQCN keep the fabric (near-)lossless.

RDMA memory vs channel semantics, and which does AI use?

Channel semantics = SEND/RECV (receiver's CPU involved, short control messages); memory semantics = READ/WRITE/atomics (access remote registered memory without the remote CPU, bulk transfers). AI/ML uses memory-semantic READ/WRITE for GPU-to-GPU, keeping the remote CPU off the path.

Why can't you troubleshoot RoCE with tcpdump on the server, and what replaces it?

RDMA bypasses the host kernel, so host capture sees nothing unless the NIC mirrors it - visibility must be designed in via NIC counters/capture flags and fabric-side flow telemetry (ECN marks, per-CoS PFC counters).

Sources

  • A. Iazzag, How does RoCE actually deliver a lossless network? (Bits & Bytes in Plain English, Apr 2026; user-supplied PDF) - parity pass 2026-07-25, CNP counter fold; configuration walkthrough excluded by invariant.
  • J. Tissieres (jtnetwork.io), RoCEv2 on Cisco Nexus 9300 and NVMe, NVMe-oF and RDMA for network engineers - PSN/NACK recovery nuance, kernel-bypass capture blind spot; full-text pass 2026-07-25.
  • IBTA RoCE specs; Nvidia GPUDirect/NCCL docs; Cisco AI/ML data-center blueprint.

domain: AI-Infra · blueprint-ref: AI-Infra 2.7 Application-level protocols (RDMA, RoCE/RoCEv2) · type: concept · status: complete · tags: [elective/ai-infra, ai/network, tradeoff/latency]

Read more