Core: Data Plane Transport

MTU, fragmentation, and PMTUD

Miro Kosut

Miro Kosut

— 2 min read
Share

Requirement / business driver

Move full-size packets end to end without silent black-holing - the classic, hard-to-diagnose failure when MTU and overlays collide.

The mechanics

  • IPv4 can fragment in transit (DF bit aside); IPv6 does NOT fragment in transit - only the source may, guided by PMTUD.
  • PMTUD relies on ICMP "packet too big". If a firewall/ACL filters ICMP, PMTUD breaks and large packets vanish - the ICMP black-hole: small packets and the handshake work, big transfers hang.
  • Encapsulation eats MTU: VXLAN/MPLS/GRE/IPsec each add overhead (Overlays and BGP EVPN, Tunneling technology selection). Either raise the underlay MTU (jumbo, Ethernet as transport) or clamp TCP MSS at the tunnel edge.
  • The converged-pod zoning doctrine (validated FlexPod-class rule set): set the switching infrastructure to 9216 once, then zone per VLAN - 9000 end-to-end only on storage/replication VLANs that never leave the pod, and 1500 on every VLAN that exits via the uplinks (management, tenant/VM access), so no distant standard-MTU hop silently fragments or black-holes. The invariant is per-VLAN endpoint consistency - every endpoint in a VLAN carries the same MTU - while a single tagged trunk happily mixes 1500- and 9000-byte VLANs. VM NICs default to 1500 and are painful to change at scale, which is itself a reason to keep anything VM-facing at standard MTU.

Recommendation / justification

Plan MTU end to end. Prefer raising the underlay/jumbo MTU to absorb encapsulation; where you can't, MSS clamping fixes TCP (but not UDP/QUIC). Never blanket-filter ICMP - PMTUD needs "packet too big". For v6 especially, MTU discipline is mandatory because in-path fragmentation isn't available.

What would change this (mid-scenario twist)

  • Overlay/tunnel added -> account for its overhead (jumbo underlay or MSS clamp).
  • "Big transfers hang, pings fine" -> suspect filtered ICMP / PMTUD black-hole.

Validation checks

  • Is MTU consistent along the path (incl. encapsulation overhead)?
  • Is ICMP PtB permitted so PMTUD works (especially for IPv6)?

IPv6 / dual-stack note

This note is essentially an IPv6 trap: no transit fragmentation means a single MTU mismatch or filtered ICMP black-holes v6 traffic. Test v6 large-packet paths explicitly.

Spaced repetition

Describe the ICMP black-hole: [...].

Describe the ICMP black-hole: a device filters ICMP "packet too big", so PMTUD fails; the handshake and small packets work but large transfers silently hang.

The key IPv6 difference for MTU is [...] (so MTU mistakes black-hole v6).

The key IPv6 difference for MTU is no in-path fragmentation - only the source fragments, guided by PMTUD (so MTU mistakes black-hole v6).

Two ways to handle encapsulation overhead are [...] or [...] (MSS clamping won't help UDP/QUIC).

Two ways to handle encapsulation overhead are raise the underlay/jumbo MTU or clamp TCP MSS at the tunnel edge (MSS clamping won't help UDP/QUIC).

The converged-pod MTU doctrine: trunks at [...], jumbo [...], 1500 on exit VLANs - and [...] (mixed-MTU VLANs may share one trunk).

The converged-pod MTU doctrine: trunks at 9216, jumbo only on storage/replication VLANs that never leave the pod, 1500 on exit VLANs - and every endpoint in a VLAN must share the same MTU (mixed-MTU VLANs may share one trunk).

Sources

  • Cisco/NetApp/SUSE, FlexPod with SUSE Rancher for AI design guide (Dec 2023) - converged-pod MTU zoning doctrine; full 75-page pass 2026-07-26 (user-supplied PDF).
  • Cisco Press, Optimal Routing Design; RFC 8201 (IPv6 PMTUD).

domain: Core · blueprint-ref: Core 4.2 Characteristics and properties (MTU/fragmentation) · type: design-decision · status: complete · tags: [core, core/dataplane, tradeoff/operability]

Read more