AI: Network

Optics and cabling selection

Miro Kosut

Miro Kosut

— 3 min read
Share

Requirement / business driver

Interconnect GPUs, leaves, and spines at 400/800G reliably and affordably - at AI scale, optics are a major cost, power, and reliability factor (tens of thousands of links).

Options

  • DAC (passive copper) - ~1-3 m, cheapest, negligible power - intra-rack.
  • AEC (active copper) - ~to 7 m - short inter-rack.
  • AOC (active optical) - ~to 30+ m, fiber pre-terminated.
  • Pluggable transceivers + fiber (SR/DR/FR/LR) - longest reach, swappable, but priciest and highest power (DSP).
  • LPO / CPO (linear-drive / co-packaged optics) - emerging, much lower power for 800G/1.6T.

Transceivers, connectors, and breakout

Optical transceivers convert electrical<->optical and follow a form-factor/speed ladder: SFP (1G) -> SFP+ (10G) -> QSFP (and QSFP28/56/112/DD) -> OSFP / QSFP-DD800 (up to 800G). AI rates of 400/800G ride QSFP-DD or OSFP. Connectors: LC is the common small-form-factor duplex; CS (very-small-form-factor) and MPO (up to 24 fibers in one ferrule) are the high-density choices for AI/ML data centers. Breakout is a key port-economics lever - one 400G port can split into 4x100G via an MPO trunk and a breakout panel, stretching spine ports further. On copper, UTP tops out around 10G; above that, fiber is required for AI throughput/distance.

Count the terminations per node. A dense 8-GPU server lands 20+ links across three logical networks and three speed classes: 8x400G back-end (one per GPU/rail), ~2x400G front-end/storage, plus 1G BMC and 10G host-management. Optics count, breakout planning, and rack cable management therefore scale with nodes x planes, not nodes alone - and the management plane, riding cheap copper or low-speed ports on its own leaf, is the one classically forgotten in port and BOM math.

Comparison

ai-optics-matrix

Tradeoffs

Copper (DAC/AEC) is cheapest and lowest-power but short; AOC/transceivers reach further at higher cost/power; transceivers are the most flexible but a top failure source and power hog; LPO/CPO trade flexibility for big power savings at the highest speeds. At fleet scale, optics power and failure rates are first-order design concerns - a flapping link stalls collectives (Latency and the straggler problem).

Recommendation / justification

DAC within the rack, AEC/AOC for short inter-rack, pluggable transceivers (DR/FR) for structured fiber and longer leaf-spine runs; watch the power budget and FEC, and weigh LPO/CPO as speeds hit 800G+. Justify by reach tier, and by the aggregate power and reliability at scale, not just per-link price.

What would change this (mid-scenario twist)

  • 800G+ with power/thermal limits -> evaluate LPO/CPO.
  • Reliability/flap issues -> favour passive DAC where reach allows; tighten optics quality.

Validation checks

  • Is each link on the cheapest/lowest-power option that meets its reach?
  • Is the aggregate optics power within facility/thermal budget, with FEC correct?

Everything above is fabric-internal optics. When training spans sites, the DCI leg rides coherent ZR/ZR+ pluggables and DWDM - the selection logic lives in Optical transport - CWDM vs DWDM, and the sync design that sets the bandwidth requirement in Bandwidth and capacity planning.

IPv6 / dual-stack note

Optics are L1 and address-agnostic.

Spaced repetition

Match reach to cabling: intra-rack = [...]; short inter-rack = [...]; longer leaf-spine/structured fiber = [...].

Match reach to cabling: intra-rack = DAC (passive copper); short inter-rack = AEC/AOC; longer leaf-spine/structured fiber = pluggable transceivers (DR/FR/LR).

Why are optics a first-order concern at AI scale (beyond price)?

Aggregate power draw and failure rate are huge across tens of thousands of links - and a flapping link stalls collective communication.

LPO/CPO matter because they [...] (linear-drive / co-packaged, fewer/no DSP).

LPO/CPO matter because they cut optics power substantially at 800G/1.6T (linear-drive / co-packaged, fewer/no DSP).

Which optics/connectors suit 400-800G AI fabrics, and what is breakout?

QSFP-DD/OSFP transceivers reach 800G; LC is the common duplex connector while CS and MPO (up to 24 fibers) are the high-density AI choices. Breakout splits one 400G port into 4x100G (MPO trunk + panel). Copper UTP tops out ~10G - above that, fiber is required.

How many network terminations does a dense 8-GPU node present, and why care?

20+ across three planes and three speeds - 8x400G back-end, ~2x400G front-end/storage, 1G BMC plus 10G management. Optics and cabling scale with nodes x planes, and the low-speed management plane is the classic omission in port/BOM math.

Sources

  • Cisco Live BRKDCN-2677 - per-node network density (HGX reference design: three networks, three speeds).
  • Cisco AI/ML blueprint; optics vendor 400/800G references; OIF LPO/CPO.

domain: AI-Infra · blueprint-ref: AI-Infra 2.3.f Impacts of various types of optics · type: design-decision · status: complete · tags: [elective/ai-infra, ai/network, tradeoff/cost, tradeoff/power]

Read more