Bandwidth and capacity planning
Requirement / business driver
Size the back-end fabric so it never throttles the GPUs - because idle GPUs waiting on the network waste the most expensive resource in the building.
Per-GPU bandwidth and non-blocking
Modern training GPUs need 400G (moving to 800G) per GPU on the back-end, and the fabric must be non-blocking (1:1) on the rails so all GPUs can run collectives at line rate simultaneously (Rail-optimized CLOS topology). Collectives fill links; oversubscription on the back-end becomes congestion and a straggler (Latency and the straggler problem).
Where oversubscription is (and isn't) OK
- Back-end GPU fabric - non-blocking; do not oversubscribe.
- Front-end / management / storage planes - can tolerate some oversubscription (different traffic profile).
Reference oversubscription ratios
Concrete anchor: a general-purpose data centre tolerates roughly 20:1 oversubscription (far more devices than simultaneous bandwidth), but AI training typically demands 1:1 (non-blocking) because every GPU sends at line rate during synchronised collectives; 4:1 is seen on some less bandwidth-bound AI workloads. The ratio is set by the traffic pattern, not a generic rule - synchronised all-reduce (Parallelism strategies and collective traffic patterns) is why 20:1 collapses to 1:1 for training.
Storage-lane floor: the validated per-GPU anchor for front-end storage access is >=12.5 Gbps per GPU (NVIDIA's recommendation), carried on 100/200G storage NICs - shared or dedicated - so the storage lane is sized like everything else here: per-GPU rate x GPU count, then checked against the front-end oversubscription it must survive.
A hybrid node adds a second floor on the same NIC budget: once inferencing shares the box with training, the ERA guidance stacks ~25 Gbps per GPU of front-end bandwidth on top of the 12.5 Gbps storage lane - ~200 Gbps of FE capacity for an 8-GPU node - which is usually the point where a dedicated storage NIC stops being optional (Validated reference design - Cisco AI-ML lossless fabric (CVD)).
The data-platform tier adds its own anchor to the spectrum: the validated multi-rack data-lake fabric runs at 2:1 oversubscription - shuffle and replication traffic is throughput-heavy but not a synchronised collective, so it sits between the GPU fabric's 1:1 and the general-purpose ~20:1.
Capacity and scale
Drive capacity from GPU count x per-GPU bandwidth x collective pattern: GPUs per rail/leaf, spine count to stay non-blocking, and pod/super-pod growth. Single-site is the norm (latency/bandwidth); multi-site distributed training needs expensive, latency-sensitive DCI and is usually avoided unless required (Connectivity models and SDN in AI fabrics). What forces multi-site is usually power - a single campus hits its megawatt ceiling before the model stops growing - and what makes it hard is physics: fiber adds on the order of a millisecond per ~200 km one-way, against a fabric engineered in microseconds, so a synchronous collective cannot simply span sites. The workable pattern is hierarchical / asynchronous training: run tight synchronous all-reduce within each site and exchange aggregated updates across sites less frequently (or asynchronously) - which converts the DCI requirement from latency-critical into bandwidth-heavy but latency-tolerant (private-fiber DWDM territory, Optics and cabling selection).
The sync hierarchy - scaling training across sites
Amdahl's law puts a hard ceiling on synchronous scaling: the fraction of each step spent in global synchronization cannot be parallelized away, so past a point adding accelerators buys almost nothing. The design response is to shrink how often everything must agree - and to let sync frequency follow proximity. That produces a three-tier hierarchy: buildings within a campus (sub-km, near-fabric latency - synchronize every step), campuses within a region (tens of km apart, operated as one campus cluster - abundant bandwidth, higher latency, synchronize less often), and cross-region (synchronize least, or asynchronously). A quiet side benefit: fewer global syncs mean fewer windows for a transient straggler to stall everyone (Latency and the straggler problem). Sites need not be equal - scale each campus's batch size to its GPU count and merge updates with a weighted average, so asymmetric campuses load-balance naturally.
Size the inter-site links from the exchange math: time per exchange ~ model size x bytes/parameter / inter-site bandwidth - even hundreds of terabytes of weights move in about a second on multi-Pb/s DCI, which is buildable because trenching and right-of-way dominate fiber cost, so pairs are laid generously (Optical transport - CWDM vs DWDM). Latency is the one constraint you cannot buy off - which is why the hierarchy exists at all. And the cross-site leg is Ethernet + coherent DWDM even for InfiniBand back-ends: IB's metro option is niche and bandwidth-poor, so between data centers Ethernet wins by default (Fabric transport - Ethernet vs InfiniBand vs UEC).
Engineering the guaranteed inter-site number
When the DCI carries a contracted floor rather than best effort - say 8x400G installed with half of it guaranteed - engineer the guarantee instead of averaging toward it: circuit-style SR policies pin explicit end-to-end paths, WECMP spreads load evenly across the parallel segment lists, and per-segment-list validity turns failures into arithmetic - a segment list crossing a dead transit link withdraws from the spread, the survivors carry on, and the still-guaranteed number is computable rather than hash-dependent (Segment Routing). The validated stack beneath it for AI DCI: multivendor SR over MACsec-encrypted spans on coherent DCO pluggables (MACsec and secure transport, Optical transport - CWDM vs DWDM).
Recommendation / justification
Build the back-end non-blocking at the GPU's full line rate, size spines for it, and keep training single-site unless a requirement forces DCI. Justify by GPU cost: an undersized fabric strands capital in idle accelerators.
What would change this (mid-scenario twist)
- GPU upgrade (400G -> 800G) -> re-size rails/spines to stay non-blocking.
- Mandated multi-site training -> hierarchical/asynchronous training (intra-site collectives, aggregated inter-site exchange); the DCI becomes bandwidth-heavy but latency-tolerant.
Validation checks
- Is the back-end truly non-blocking at per-GPU line rate (rails and spine)?
- Is oversubscription confined to front-end/storage, never the GPU fabric?
IPv6 / dual-stack note
Capacity is address-agnostic; plan MTU/headroom identically for whichever family the fabric uses.
Related
- Rail-optimized CLOS topology
- Latency and the straggler problem
- Fabric transport - Ethernet vs InfiniBand vs UEC
- Connectivity models and SDN in AI fabrics
Spaced repetition
Why is the GPU back-end fabric sized non-blocking at per-GPU line rate?
Collectives saturate links simultaneously; oversubscription causes congestion/stragglers and strands very expensive GPUs idle.
Where is oversubscription acceptable in an AI design, and where not?
Acceptable on front-end/management/storage planes; not on the back-end GPU fabric.
Why is AI training usually single-site?
Multi-site needs expensive, latency-sensitive DCI for collectives - avoided unless a requirement forces it.
Typical oversubscription: general DC vs AI training?
General-purpose DC ~20:1; AI training ~1:1 (non-blocking) because GPUs send at line rate during synchronised collectives; ~4:1 on some less bandwidth-bound AI workloads.
Multi-site training becomes workable through [...] - synchronous collectives stay inside each site and only aggregated updates cross the DCI, turning the interconnect requirement from latency-critical into bandwidth-heavy.
Multi-site training becomes workable through hierarchical / asynchronous training - synchronous collectives stay inside each site and only aggregated updates cross the DCI, turning the interconnect requirement from latency-critical into bandwidth-heavy.
Why does multi-site training synchronize less often the farther apart the GPUs are?
Amdahl's law - the synchronized fraction of each step caps parallel speedup, so the design shrinks global-sync frequency along a hierarchy (every step within a campus, less across a campus cluster, least/async cross-region); fewer syncs also give transient stragglers fewer chances to stall everyone.
How do you size the DCI for hierarchical training, and what stays the binding constraint?
From the exchange math - model size x bytes/parameter / bandwidth per sync; bandwidth is buildable (trenching dominates fiber cost, so pairs are overprovisioned) while latency is physics - the unbuyable constraint the hierarchy is designed around.
Campuses of unequal size join one training run by scaling each site's [...] and merging updates with a weighted average.
Campuses of unequal size join one training run by scaling each site's batch size to its GPU count and merging updates with a weighted average.
A hybrid training-plus-inference node stacks [...] on the 12.5 Gbps storage floor - about 200 Gbps of FE capacity for an 8-GPU server.
A hybrid training-plus-inference node stacks ~25 Gbps per GPU of front-end bandwidth on the 12.5 Gbps storage floor - about 200 Gbps of FE capacity for an 8-GPU server.
Sources
- Cisco AI POD for Enterprise Training and Fine-Tuning Design Guide (Jan 2026) - 25 Gbps/GPU hybrid-inference FE floor on top of the storage lane; full 94-page pass 2026-07-26 (user-supplied PDF).
- Cisco Live BRKSP-2551 (2026), Introduction to Segment Routing - A foundation for Autonomous Networks (Donzelli, Cisco / Gorovoy, Nebius; incl. the Microsoft AI-fabric and Nebius DCI material)
- Cisco, CDIP Cloudera GenAI CVD (May 2024) - validated 2:1 oversubscription for the multi-rack data-platform tier; passed 2026-07-25.
- Cisco AI POD for Enterprise Training and Fine-Tuning Design Guide (CVD, Jan 2026) - 12.5 Gbps/GPU storage-lane anchor
- Cisco AI/ML blueprint; Nvidia reference architectures (per-GPU bandwidth, non-blocking).