AI: Compute, storage & environment

Scale-up vs scale-out

Miro Kosut

Miro Kosut

— 4 min read
Share

Requirement / business driver

Connect many GPUs so collectives run fast - across two dimensions whose boundary shapes the whole fabric.

The two dimensions

  • Scale-up (inside the node/domain) - NVLink / NVSwitch (and the emerging UALink - an open, vendor-neutral consortium standard; AMD's Infinity Fabric is the proprietary parallel) connect GPUs within a server or tightly-coupled domain at very high bandwidth, low latency, memory-semantic access. Highest bandwidth, smallest scope (e.g., 8 GPUs, or larger NVLink domains).
  • Scale-out (across nodes) - the RDMA Ethernet/InfiniBand network (Fabric transport - Ethernet vs InfiniBand vs UEC, Rail-optimized CLOS topology) connects nodes and pods. Larger scope, lower per-link bandwidth than NVLink, and where contention/stragglers live.

The design lever

Collective libraries are topology-aware - they use NVLink first (intra-node), then the network. So a bigger scale-up domain means less scale-out traffic, which means a smaller, cheaper, less-contended fabric for the same job. The scale-up/scale-out boundary is therefore a primary design decision.

Between a full NVSwitch domain (any-to-any among 8 GPUs) and plain scale-out sits the PCIe-platform reality: NVLink bridges. PCIe cards of the H200 NVL class interconnect only in fixed 2-way or 4-way bridge groups (NVL2/NVL4), and inference-class cards (L40S / RTX-Pro class) carry no NVLink at all, leaving peer traffic on PCIe. Two design consequences follow. Parallelism placement: a tensor-parallel group must fit inside the bridge domain - pair or quad - or its traffic drops onto PCIe and the fabric (Parallelism strategies and collective traffic patterns). Population rules are a design check, not assembly trivia: the validated design warns that GPUs placed across different PCIe switches route peer traffic through the CPUs, so which slots a bridge group occupies decides the bandwidth it actually gets (AI-enabling hardware - GPU, DPU, SmartNIC).

Tradeoffs

Scale-up is fastest but bounded by hardware, cost, power, and cooling (Sustainability - Green AI, power and cooling); scale-out is flexible and effectively unbounded but slower per link and contention-prone.

The third axis: scale-across

Industry usage has added a third named dimension. Scale-across extends GPU-level communication between clusters or sites, treating separate facilities as one logical training system - long-haul RDMA over DCI with controlled jitter, because the synchronisation that collectives demand does not relax just because the fibre got longer. Candidate transports are RoCEv2 over dedicated DWDM DCI and, prospectively, Ultra Ethernet profiles aimed at distance. The feasibility doctrine already lives in Bandwidth and capacity planning: what forces multi-site is usually the power ceiling of one campus, physics adds about a millisecond per ~200 km, and the workable pattern is hierarchical/asynchronous training that keeps tight synchronous collectives inside each site - converting the inter-site leg into bandwidth-heavy but latency-tolerant traffic. Read the three axes as one budget: scale-up is invisible to the fabric yet shapes it (a tightly coupled accelerator group presents its aggregate demand as 4-8+ high-speed NICs per host at the ToR, and rack power density climbs toward the megawatt, Sustainability - Green AI, power and cooling), scale-out is the fabric itself, and scale-across is the admission that no single facility is the ceiling.

Recommendation / justification

Use the largest practical scale-up domain (NVLink/NVSwitch) for tightly-coupled GPUs, then scale out over a rail-optimized non-blocking fabric. Size the split by the job's model/data parallelism. Justify by the fact that maximizing scale-up directly shrinks the network burden and improves JCT (Latency and the straggler problem).

What would change this (mid-scenario twist)

  • Larger NVLink domains available -> push more traffic to scale-up, shrink the fabric.
  • Job needs more GPUs than one scale-up domain -> scale out, size the fabric non-blocking (Bandwidth and capacity planning).

Validation checks

  • Is tightly-coupled traffic kept on scale-up (NVLink) where possible?
  • Is the scale-out fabric non-blocking for the GPUs that must cross it?

IPv6 / dual-stack note

Scale-up is a hardware interconnect (address-agnostic); scale-out routing can be dual-stack.

Spaced repetition

Scale-up vs scale-out: scale-up = [...]; scale-out = [...].

Scale-up vs scale-out: scale-up = NVLink/NVSwitch within a node/domain (highest BW, memory-semantic, small scope); scale-out = RDMA network across nodes (flexible, larger scope, lower per-link BW).

Why does a bigger scale-up domain help the network?

Collectives use NVLink first, so more intra-domain traffic means less scale-out traffic - a smaller, cheaper, less-contended fabric for the same job.

The scale-up domain is bounded by [...]; scale-out is bounded by [...].

The scale-up domain is bounded by hardware, cost, power, and cooling; scale-out is bounded by per-link bandwidth and contention.

Scale-up, scale-out, scale-across - one line each?

Scale-up = accelerator interconnect inside a node/rack (NVLink/NVSwitch, UALink); scale-out = the RDMA fabric across nodes in a cluster; scale-across = GPU communication between clusters/sites over long-haul, jitter-controlled RDMA DCI, treating facilities as one logical system.

The third scaling axis - GPU communication between clusters or sites over long-haul RDMA - is [...].

The third scaling axis - GPU communication between clusters or sites over long-haul RDMA - is scale-across.

On PCIe GPU platforms the scale-up domain is a [...] (or nothing at all on L40S / RTX-class cards) - so tensor-parallel groups must fit the bridge domain or their traffic falls onto PCIe and the fabric.

On PCIe GPU platforms the scale-up domain is a 2- or 4-way NVLink bridge group (or nothing at all on L40S / RTX-class cards) - so tensor-parallel groups must fit the bridge domain or their traffic falls onto PCIe and the fabric.

Sources

  • Cisco AI POD for Enterprise Training and Fine-Tuning Design Guide (CVD, Jan 2026) - bridged NVLink tier and PCIe population rules
  • J. Tissieres (jtnetwork.io), Understanding Scale-Out, Scale-Up, and Scale-Across Networking (2026); full-text pass 2026-07-25.
  • Nvidia NVLink/NVSwitch (and UALink) references; Cisco AI/ML blueprint.

domain: AI-Infra · blueprint-ref: AI-Infra 4.1 Compute resources (scale-up vs scale-out) · type: design-decision · status: complete · tags: [elective/ai-infra, ai/hardware, tradeoff/scale, tradeoff/latency]

Read more