Miro Kosut

AI: AI-ML fundamentals & governance

Service placement - on-prem, cloud, hybrid, distributed

Requirement / business driver Decide where AI workloads run, balancing data gravity, cost/ROI, GPU availability, latency, and compliance. Functional vs non-functional requirements Placement decisions hang on two requirement types. Functional requirements define what the system does - scalability, integration with existing infrastructure/services, user experience, security capability. Non-functional requirements

AI: AI-ML fundamentals & governance

Data sovereignty and data gravity

In one line Data-residency law and the sheer mass of training data ("data gravity") frequently dictate where AI infrastructure must sit - often overriding cost or elasticity preferences. Data sovereignty Residency / localization rules require data to stay in-country or in-region; cross-border transfer is restricted; sovereign-

AI: AI-ML fundamentals & governance

Regulations, governance, and AI security policy

Requirement / business driver Meet regulatory and governance obligations and manage the new corporate-security-policy risks that AI introduces. AI governance Model and data governance - lineage, versioning, auditability; assurance and integrity (the model behaves as intended; training/inference data is trustworthy and unaltered); and access control to models and datasets.

AI: AI-ML fundamentals & governance

Cost, ROI, and scalability of AI infrastructure

In one line GPUs are extraordinarily expensive, so cost/ROI and utilization dominate AI-infra decisions - and training vs inference scale very differently. Cost CapEx (own GPUs + power + cooling) vs OpEx (cloud rental). GPUs dominate TCO, and power/cooling (Sustainability - Green AI, power and cooling) is a major ongoing cost

AI: AI-ML fundamentals & governance

Workload placement, mobility, and vendor lock-in

Where an AI workload runs - and how freely it can move - is a design and business decision, not just an ops one. This note covers mobility (moving workloads across DC/cloud/edge) and interoperability / vendor lock-in (keeping that freedom). It builds on the static placement trade in Service placement

AI: Network

RDMA, RoCE and RoCEv2

In one line RDMA lets a NIC read/write remote memory directly - bypassing the CPU and kernel - which is what makes GPU-to-GPU communication fast enough for training; RoCE carries RDMA over Ethernet. RDMA Zero-copy, kernel-bypass, NIC-offloaded transfers via the verbs API: microsecond latency, line-rate

AI: Network

Fabric transport - Ethernet (RoCEv2) vs InfiniBand vs UEC

The signature AI-Infra design decision: what transport carries the backend / scale-out GPU fabric (the east-west, RDMA, highest-speed network that GPUs use for collective communication). Get this wrong and every other choice inherits the pain. Requirement / business driver Move synchronized collective traffic (all-reduce / reduce-scatter) between

AI: Network

Lossless fabric - PFC, ECN, DCQCN

In one line The set of mechanisms that make an Ethernet fabric near-lossless so RoCEv2 RDMA can run without the retransmits that would wreck Job Completion Time. How it works * ECN (Explicit Congestion Notification) - end-to-end, early. A congested switch marks the IP ECN bits; the receiver returns

AI: Network

Rail-optimized CLOS topology

Requirement / business driver Build a non-blocking GPU back-end fabric whose topology minimises hops and contention for collective communication - because the topology directly sets training performance. CLOS / spine-leaf baseline Leaf (ToR) + spine, non-blocking via ECMP, scaled by adding spines and pods. Every leaf reaches every other through

AI: Network

Optics and cabling selection

Requirement / business driver Interconnect GPUs, leaves, and spines at 400/800G reliably and affordably - at AI scale, optics are a major cost, power, and reliability factor (tens of thousands of links). Options * DAC (passive copper) - ~1-3 m, cheapest, negligible power - intra-rack. * AEC (active copper) - ~to 7 m - short

AI: Network

Latency and the straggler problem

In one line Distributed training is barrier-synchronised, so the slowest path or GPU (the straggler) gates the entire step - which is why tail latency, not average, governs Job Completion Time (JCT). Why one slow link stalls thousands of GPUs Collectives (all-reduce, all-gather) require every GPU to finish

AI: Network

Bandwidth and capacity planning

Requirement / business driver Size the back-end fabric so it never throttles the GPUs - because idle GPUs waiting on the network waste the most expensive resource in the building. Per-GPU bandwidth and non-blocking Modern training GPUs need 400G (moving to 800G) per GPU on the back-end, and