Fabric resiliency and failure handling

A training job spreads across hundreds of GPUs, so the fabric must keep running through device failures. The good news of scale: *the bigger the fabric, the higher the chance something fails - but the smaller each failure's blast radius.*

Spine failure - graded, not catastrophic

In a non-blocking spine-leaf fabric, losing one spine removes a fraction of fabric bandwidth equal to 1 / (number of spines): a 4-spine fabric drops ~25%, an 8-spine fabric ~12.5%, and so on. Traffic keeps flowing on the survivors at reduced capacity. So adding spines does double duty - more bisection bandwidth (Rail-optimized CLOS topology) and a smaller per-failure hit. This resilience matters most during inference, where availability is the priority.

Leaf failure - dual-homing saves the host

Hosts are dual-homed to two leaf switches (Validated reference design - Cisco AI-ML lossless fabric (CVD)), so a leaf failure does not disconnect them - it halves their bandwidth until repair. A leaf with no redundancy would strand every host attached to it.

What happens to in-flight traffic

Flows crossing the failed device lose packets that must be retransmitted, so affected AI jobs slow down briefly but usually don't fail. Two design choices cut that pain:

• Hardware-based failure detection reacts faster than software, minimising packet loss and avoiding blackholing (sending traffic onto a link that is already down).
• Selective retransmission (in modern Scheduled Ethernet fabrics) resends only the lost packets - far more efficient than the classic go-back-N of legacy RoCE, which rewinds and resends everything after the loss (RDMA, RoCE and RoCEv2).

Silent failures: SDC, burn-in, and the checkpoint blast radius

Synchronous training has cruel failure arithmetic: one failed GPU in a 100k-GPU job sends the entire job back to its last checkpoint unless the training stack tolerates partial failure - and a fleet limping at reduced MFU from accumulating stragglers (Latency and the straggler problem) burns capex just as surely. The nastiest class is Silent Data Corruption (SDC): hardware quietly produces wrong results with no alert - surfacing, if at all, as NaNs or exploding gradients. Detection is a designed-in discipline, not an afterthought: watch gradient norms for anomalies (imperfect - spikes also come from data batches and hyperparameters), and run diagnostic sweeps as prologue and epilogue around every large job, because standard GPU diagnostics miss corner cases - even a switch ASIC's arithmetic unit has silently corrupted in-network all-reduce results while passing health checks. The infrastructure countermeasure is burn-in discipline: component failures follow a bathtub curve, so serious operators stress-cycle new clusters at temperature until infant-mortality failures are burned out before production - with optical transceivers the top wear item under thermal cycling (Optics and cabling selection).

Site- and cluster-level DR

Intra-fabric redundancy (above) keeps a job running through component loss; DR asks what happens when the cluster or the site is lost. For training, the recovery unit is the checkpoint: RPO = checkpoint cadence (GPU-hours of work you accept losing) and RTO = restore + re-provision time, so AI DR is mostly a data-and-checkpoint problem (Storage and checkpointing for AI), not a fabric one. The design levers: where standby capacity lives - a second owned cluster is idle capital, while cloud burst trades capex for egress cost, lock-in, and sovereignty exposure (Workload placement, mobility, and vendor lock-in, Data sovereignty and data gravity); what gets replicated - datasets, checkpoints, container images/registry, and orchestration state, sized against inter-site bandwidth on the front-end/WAN path (never the back-end fabric); and tiering by workload - inference is near-stateless and fails over like any service (GSLB/anycast, N+1 capacity), while training normally accepts restore-and-restart because synchronous cross-site training is impractical - collectives are latency-bound (Latency and the straggler problem). State RPO in GPU-hours and validate the restore path end to end, including the time to re-form the standby fabric's lossless state.

Recommendation / justification

Size the spine count for both bandwidth and acceptable single-spine loss; dual-home every host; and prefer NICs/fabrics with hardware failure detection and selective retransmission. Justify by job survivability and tail latency - a resilient fabric turns a failure into a slowdown, not a restart of a multi-hour training run.

What would change this (mid-scenario twist)

• Inference-heavy / HA requirement -> add spines for redundancy, not just bandwidth.
• Legacy go-back-N NICs -> a single drop is far costlier; tighten losslessness (Lossless fabric - PFC, ECN, DCQCN) or move to selective-retransmission-capable hardware.
• Single-homed hosts found in an audit -> flag as a fate-sharing risk.

IPv6 / dual-stack note

Failure detection (BFD) and the redundant paths must protect IPv4 and IPv6 equally; a v6 path with no fast detection reintroduces blackholing for half the stack.

Related

• Rail-optimized CLOS topology
• Validated reference design - Cisco AI-ML lossless fabric (CVD)
• Latency and the straggler problem
• Load balancing in AI fabrics - flow, flowlet, packet spray

Spaced repetition

Sources

• Cisco U. Key Network Challenges and Requirements for AI Workloads - Redundancy and Resiliency Considerations.

domain: AI Infrastructure · type: design-note · status: complete · tags: · source: Cisco U. - Key Network Challenges and Requirements for AI Workloads (Redundancy and Resiliency)