Storage networking - FC, FCoE, iSCSI, and NVMe-oF

In one line

Storage networks are designed around one question - how does the fabric guarantee delivery, and who pays for that guarantee - and each protocol answers it differently: FC builds losslessness into the link layer, FCoE borrows it from Ethernet's DCB toolkit, iSCSI lets TCP absorb loss, and NVMe-oF inherits whichever transport it rides.

The FC design canon

Fibre Channel is a purpose-built SAN stack. Devices log in to the fabric (FLOGI) and are known by WWPNs registered in the fabric-wide name server; routing between switches is FSPF, a link-state protocol. Losslessness comes from buffer-to-buffer credits - a transmitter may only send when the next hop has explicitly granted a buffer - so frames are never dropped by construction, the philosophical opposite of Ethernet's mark-then-pause model (Lossless fabric - PFC, ECN, DCQCN). Access control is zoning (who may talk to whom; single-initiator zoning is the default discipline), and VSANs are the SAN's VRF - independent virtual fabrics, each with its own zoning and FSPF, on shared switches. Topologies follow core-edge (or collapsed-core at small scale) with engineered fan-in/oversubscription ratios from host edge to storage core. The availability doctrine is the part people miss: production SANs run two fully independent fabrics (A and B) - separate switches, no ISLs between them - and host multipathing (MPIO) provides failover across them, so any fabric-wide state accident (zoning push, FSPF event, code upgrade) is contained to one fabric while I/O continues on the other.

Three fabric services carry the converged validations and belong in the canon: NPIV lets one physical N-port hold many fabric logins - it is how every vHBA behind a fabric interconnect appears to the SAN; F-port-channel-trunks virtualise those NPV-edge logins over a resilient port-channel, so a member-link failure moves sessions instead of dropping them; and smart zoning with device aliases keeps zoning honest at scale - initiator/target-aware zones cut TCAM burn, and alias indirection turns a WWPN change into a one-line edit instead of a zone-set rewrite.

The SAN tier itself has a collapse option the canon should name: current Nexus datacentre switches can run SAN switching natively - native FC ports and 100G FCoE port-channels from the fabric interconnects landing in the same box that switches Ethernet - trading the dedicated MDS tier and its deeper SAN-analytics tooling for fewer devices and one fabric team; every FC service above applies unchanged.

The isolation VSANs buy is deeper than a shared-switch convenience: separation is hardware-enforced, and every fabric service runs independently per VSAN - zone server, name server, login server - so RSCNs and fabric events are contained inside their VSAN rather than rippling across the physical estate; that containment, not port savings, is the high-availability argument. Two stability rules ride along: give every switch a static domain ID (a fabric reconfiguration can otherwise reallocate FCIDs under live devices), and where fabric-services distribution must reach NPV edges over IP multicast, keep the multicast address unique per fabric - a shared address quietly cross-joins fabrics' distribution planes.

Zoning has an arithmetic the canon should carry. A standard zone makes every member an ACL peer of every other, so a zone of n members burns about n x (n-1) TCAM entries - which is why the discipline is 1:1 zones or, better, smart zoning, whose initiator/target attributes program only I-T pairs and support large zones (hundreds of members) at a fraction of the TCAM. The multiplier people miss: on an F-port-channel the zoning TCAM is programmed on every member link, so a fabric-interconnect uplink channel multiplies the burn by its member count - and if TCAM runs hot (alerting fires around 80% of a region or 60% total), splitting one big F-port-channel into two halves the per-member load because the logins split between them.

The ISL doctrine is a short list with sharp reasons. Port-channels aggregate up to 16 members; hashing is per-exchange (source, destination, exchange ID), and a member failure is invisible to FSPF - bandwidth shrinks, the routing table never churns - which is the availability argument for channels over parallel single ISLs. Keep ISL speed at or above edge speed (a slower ISL is a built-in congestion point), remember only equal-cost channels share load (FSPF cost follows bandwidth), and note that members may take different physical paths with different latencies - DWDM spans included. Two habits complete it: run two parallel port-channels between switch pairs so one can be drained, upgraded, or re-VSANed while the other carries traffic, and spread members across line cards and forwarding engines - an eight-module spread turns a module failure into a 12.5% capacity event instead of 50%.

FCoE - the convergence trade

FCoE carries FC frames inside Ethernet, which only works if that Ethernet class never drops: PFC makes the storage class lossless, ETS guarantees it bandwidth, and DCBX negotiates both - the very same DCB toolkit RoCEv2 later leaned on. FIP handles discovery and fabric login across the Ethernet segment, and hosts use CNAs carrying LAN and SAN on one wire. The pay-off is convergence at the access layer - fewer adapters, cables, and switches - which is why single-hop FCoE (server to access switch, then native FC or FCoE northbound within one hop) was the design that earned its keep, while multi-hop FCoE rarely justified coupling the whole storage path to Ethernet operations. Treat FCoE today as a brownfield convergence tool, not a target architecture.

One converged-access collision is worth knowing by name: FCoE's de-facto no-drop class is CoS 3, and so - for two decades - is call signaling (CS3). Where UC and FCoE share converged links, the design must pick a tactic before it bites: dedicate CoS 3 to FCoE and carry signaling on a different L2 marking in the data center (restoring it at the boundary), or keep FCoE and UC traffic on separate infrastructure/classes so the overlap never materializes. The point is recognition - the collision is silent until signaling starts sharing a lossless class's fate. AVB media then makes it a triple booking: SR class A also rides CoS 3 (PTP timing for AI fabrics).

iSCSI - block storage over commodity IP

iSCSI encapsulates SCSI in TCP, so it needs nothing special from the fabric - loss is retransmitted, at latency cost. Its design rules are correspondingly simple: a dedicated storage VLAN/segment, jumbo MTU end-to-end (verify it - see MTU, fragmentation, and PMTUD), MPIO across two paths rather than LACP (multipathing is per-session and storage-aware; a port-channel is neither), and CHAP for initiator authentication. It is the cheapest block-storage entry and perfectly adequate for management tiers and general workloads - it is not the training data path.

The converged-worker corollary (validated in the FlexPod RAG design): pick the HA mechanism per traffic class. Storage vNICs are built one per fabric with fabric failover disabled - MPIO owns recovery, and a failover mechanism underneath would fight it - while management and NFS vNICs, which have no multipath layer, use fabric failover as their single-path safety net. The reason the failover lives in the fabric rather than in host bonding is operational: a per-vNIC failover policy in the profile is set once and inherited by every server the policy touches - transparent to the OS, nothing per-host to configure or drift - while OS teaming's historical motive, aggregating slow links, is moot at 25/100G lane speeds. The same per-fabric lane pair can carry NVMe-TCP as tagged VLANs on the iSCSI vNICs - two block protocols, one isolation scheme, still MPIO-recovered. The hyperconverged cousin of the same decision: on FI-attached HCI (AHV class) the hypervisor bond runs active-passive - LACP is unsupported in managed mode - so a host's usable bandwidth is one fabric's worth, and the A/B fabrics must be planned as a failover pair, not an aggregate. The FlexPod bare-metal validation states the same constraint generically - the FIs do not support LACP NIC teaming - so per-fabric bandwidth planning is the rule on any FI-attached host; vault-carried on second confirmation.

SAN boot and the stateless server

Booting hosts from array-hosted LUNs instead of local disk is what makes converged compute genuinely stateless: the server's whole identity - WWPNs, MACs, UUID, boot policy - lives in the profile, and its whole state lives on the array, so the sheet metal becomes fungible. A failed node is recovered by associating the profile with spare hardware and booting the same LUN - no reinstall, no data migration - and the same move repurposes hardware between roles or tenants (the per-tenant boot-SVM fork in Tenant isolation and segmentation in shared AI clusters builds directly on this). Secondary wins: diskless nodes (less to buy, less to fail) and a DR story - replicate boot LUNs alongside data and the fleet's identity travels with them. The boot itself stays trustworthy via the secure/measured-boot triad in AI-enabling hardware - GPU, DPU, SmartNIC.

NVMe-oF - the flash-era fabric protocol

NVMe-oF extends NVMe's parallel queue model end-to-end, removing the SCSI translation that made older protocols a bottleneck for flash. The design decision is the transport: FC-NVMe rides an existing FC estate (the natural brownfield modernization), NVMe/RoCE is fastest but imports the full lossless-Ethernet operating model (Lossless fabric - PFC, ECN, DCQCN), and NVMe/TCP runs on any IP fabric with a modest latency premium - the same performance-versus-operability fork as Fabric transport - Ethernet vs InfiniBand vs UEC, one layer up. In AI infrastructure, NVMe-oF serves the checkpoint/flash tier, while bulk dataset delivery usually belongs to parallel filesystems or object storage - the strategy split lives in Storage and checkpointing for AI.

The scale of the shift is visible in the queue model: SCSI serialised a device behind one queue of 64 commands, while NVMe exposes up to 64K queues of 64K commands each, mapped per CPU core and NUMA-aware - which is why flash stopped being the bottleneck and the network inherited the job. Between RoCE and TCP sits a fourth transport branch worth naming: NVMe/iWARP runs RDMA over TCP, keeping the NIC offload while tolerating loss - no lossless class to operate - but its NIC ecosystem is small and, despite sharing the verbs API, iWARP and RoCE endpoints cannot interoperate (different L3/L4 machinery), so the choice is fabric-wide.

Fabric health - the eviction doctrine and the congestion counters

Mature FC fabrics automate a stance the Ethernet world is still learning: a component that corrupts or stalls traffic is removed from service, automatically. Internally, frames can be corrupted at five stages between a switch's ingress and egress buffers; the platform doctrine is to detect at each stage and power down the failing module or crossbar rather than tolerate silent corruption (log-only modes exist, but eviction is the recommended default). At the port edge the same stance is port-guard: error-disable ports on link-integrity symptoms - link, sync, or signal loss, invalid CRC, input errors - at both core and edge, but error-disable on congestion symptoms only at the edge: a slow-draining host deserves eviction, a core ISL almost never does (credit-loss recovery being the one exception). The counter language beneath it: tx-wait - the fraction of time a port sits at zero transmit credits with frames queued - is the primary truth-counter for FC congestion, with a severity ladder above it (credit-loss recovery and link resets at the severe end, timeout discards next, credit-not-available and slow-port delay as early warning), and per-hop path tracing exposes TxWait and credit state along an entire initiator-target path. Two link-integrity services keep the substrate honest: FEC, on by default at current speeds, corrects wire errors at near-zero latency - and its corrected-blocks counter is free early-warning telemetry, since rising corrections mean a degrading link before anything drops - and B2B credit recovery (BB_SC_N) restores credits silently leaked when corruption eats a start-of-frame or an R_RDY; it is negotiated at login, so the HBA must support it too. The cheapest rule closes the list: shut unused ports - less churn, no surprise logins, and on current hardware the transceiver powers off, saving roughly a watt per port. The stance rhymes with Fabric resiliency and failure handling: detect fast, evict the sick component, let redundancy carry.

Securing the fabric - beyond zoning

Zoning is access control, not security. The FC ladder above it: Fabric Binding allowlists which switches may join the fabric (an unlisted switch's ISL simply never initialises); port-security allowlists which devices may log in, and where (with auto-learn to seed the database); FC-SP (DH-CHAP) authenticates links with modern hashes; and TrustSec FC link encryption protects ISLs in flight. The sleeper every design review should ask about: inband FC management is open by default - an attached host can query the name server and, via zone generic services, even push zoning changes unless GS is set read-only and FC-CT management security restricts who may ask. On the management plane the modern row reads like any hardened platform - pruned crypto algorithms, current-generation SNMP and AAA transports, certificate automation, and identity-provider authentication (MFA/SSO) that keeps credentials off the switch entirely - with FIPS-validated builds where compliance demands.

Comparison

Recommendation / justification

For a greenfield AI build, keep storage on IP/Ethernet - NVMe-oF (TCP first, RoCE where the lossless discipline already exists for the GPU fabric) plus a parallel FS/object tier - on a dedicated storage network, and skip FC entirely: one skill set, one vendor-open transport, no second fabric type. FC stays where a brownfield estate, regulated data, or an MDS operations team already exists - modernize it via FC-NVMe rather than rip-and-replace. FCoE is defensible only as single-hop access convergence in that brownfield. iSCSI covers cheap general block. Justify the choice by operating model and loss-guarantee ownership, not by peak numbers.

What would change this (mid-scenario twist)

• Existing MDS/FC estate and SAN team -> FC-NVMe modernization path, not IP migration.
• Mandate to converge cabling at the access layer -> single-hop FCoE with strict DCB discipline.
• No appetite to operate a lossless class -> NVMe/TCP everywhere; accept the latency premium.
• Dataset throughput dominates -> shift spend to the parallel-FS tier (Storage and checkpointing for AI).

Validation checks

• Are fabrics A and B truly independent (no inter-fabric ISLs), and does pulling one path prove MPIO failover?
• Is single-initiator zoning (or peer zoning) enforced, with no open zones?
• Where FCoE or NVMe/RoCE runs: is the no-drop class proven (PFC counters, watchdog armed), not assumed?
• Is jumbo MTU verified end-to-end on every iSCSI/NVMe-TCP segment?

• Can you trace a host-to-target path hop by hop and read TxWait/credit state per member - and does tx-wait sit near zero on core ISLs under load?

IPv6 / dual-stack note

FC and FCoE have no IP layer, so dual-stack does not apply on the data path (management interfaces still do). iSCSI and NVMe/TCP are fully dual-stack - align them with the site's v6 plan - and NVMe/RoCE follows RoCEv2's dual-stack behaviour (RDMA, RoCE and RoCEv2).

Related

• Storage and checkpointing for AI - where each protocol fits the AI data strategy
• Lossless fabric - PFC, ECN, DCQCN - the DCB toolkit FCoE and NVMe/RoCE depend on
• Fabric transport - Ethernet vs InfiniBand vs UEC - the same performance-vs-operability fork
• RDMA, RoCE and RoCEv2 - the RDMA transport under NVMe/RoCE
• PTP timing for AI fabrics - AVB SR class A is CoS 3's third claimant

Spaced repetition

Sources

• Cisco Live BRKDCN-2941 (E. Mazurek), Advanced Storage Area Network Design (2026 deck, user-supplied PDF; all 105 pages passed 2026-07-26) - VSAN service isolation, zoning TCAM arithmetic, ISL/port-channel doctrine, eviction/health canon, FC security ladder.
• Cisco/NetApp/SUSE, FlexPod with SUSE Rancher for AI design guide (Dec 2023) - Nexus native SAN-switching collapse option; SAN-boot/stateless-profile rationale, fabric-failover-vs-bonding argument; full 75-page pass 2026-07-26 (user-supplied PDF).
• Cisco/Hitachi Adaptive Solutions OCP AI CVD (Oct 2024) - MDS canon services (NPIV, F-port-channel-trunk, smart zoning/device alias); Cisco/NetApp FlexPod OpenShift AI MLOps CVD (May 2025) - FI no-LACP second confirmation; passed 2026-07-25.
• Cisco, Compute Hyperconverged with Nutanix GPT-in-a-Box 1.0/2.0 CVDs (2024/2025) - active-passive hypervisor bond, no LACP in managed mode; design chapters passed 2026-07-25.
• Cisco, FlexPod for Accelerated RAG Pipeline with NVIDIA NIM and Cisco Webex CVD (Oct 2024) - worker vNIC HA split, NVMe-TCP tagged-VLAN lanes; passed through the Solution Design chapter 2026-07-25 (benchmarking and Webex chapters not captured).
• J. Tissieres (jtnetwork.io), NVMe, NVMe over Fabrics and RDMA for network engineers - queue-model magnitudes, NVMe/iWARP branch, lossy-tolerance per transport; full-text pass 2026-07-25.
• Cisco Press, End-to-End QoS Network Design, 2nd Ed. (Szigeti et al.) - CoS 3 overlap.
• Cisco Live BRKENS-2094 - AVB SR class A / CoS 3.
• Cisco SAN design canon (MDS 9000 / UCS design guidance), paraphrased
• Storage Networking Protocol Fundamentals (James Long) - queued for parity triage

domain: AI Infrastructure · type: design-note · status: complete · tags: [ai-infrastructure, compute-storage] · blueprint-ref: AI-Infra 4.4.b (FC, FCoE, NVMe, NVMe-oF, software-defined storage) · source: Cisco SAN design canon (MDS/UCS design guides), paraphrased from working knowledge