AI: AI-ML fundamentals & governance

Workload placement, mobility, and vendor lock-in

Miro Kosut

Miro Kosut

— 4 min read
Share

Where an AI workload runs - and how freely it can move - is a design and business decision, not just an ops one. This note covers mobility (moving workloads across DC/cloud/edge) and interoperability / vendor lock-in (keeping that freedom). It builds on the static placement trade in Service placement - on-prem, cloud, hybrid, distributed.

What actually moves

AI workload mobility means moving one of three things: the model, the inference application, or the data it consumes/produces. The two halves of the workload place very differently:

  • Training is resource-intensive and latency-coupled -> stays in a central, non-blocking cluster (on-prem or a cloud region).
  • Inference is lighter and latency-to-user-sensitive -> a strong candidate for the edge, moving toward demand.

Why move workloads

Cost (cheaper power/compute, off-peak, price arbitrage across clouds - and avoiding cloud lock-in); DR and load-balancing (shift on failure, balance across sites, follow business hours across time zones); performance (closer to users = lower latency); compliance (keep data in a permitted jurisdiction - Data sovereignty and data gravity); optimisation (place each job where it fits best); multi-tenancy (carve fabric resources per business unit - Tenant isolation and segmentation in shared AI clusters).

How workloads move

  • Manual - control at the cost of effort/error: SCP / cloud object storage (S3), container moves (Docker/Rancher/K8s), hypervisor vMotion/snapshots, pulling models from Git.
  • Automated - efficiency at the cost of tooling/expertise: Kubernetes (+ Kubeflow for ML, OpenShift, Mesos) for containerised jobs; Airflow/Spark for the data/ETL pipeline; hybrid managers (Tanzu, Anthos, Azure Arc); FaaS for event-triggered inference; AIOps platforms that move workloads on live performance signals; model servers (TensorFlow Serving, TorchServe).
  • The network is the enabler. Mobility only works if services follow the workload: VXLAN EVPN for software-defined L2/L3 reachability and SD-WAN for secure, prioritised flows across geographies (Connectivity models and SDN in AI fabrics).

Vendor lock-in (and how to design against it)

Lock-in is overreliance on one vendor's hardware, software, tools, or cloud such that switching costs (money, disruption, data) become prohibitive. In AI the classic example is NVIDIA + InfiniBand: so dominant in HPC/AI interconnect that it behaves as a de-facto proprietary standard (a core reason Ethernet/UEC matters as the open alternative - Fabric transport - Ethernet vs InfiniBand vs UEC). Dependencies hide in four places: technology/integration, data/API formats, human capital (skills tied to one stack), and long-term contracts.

Mitigation is a proactive strategy:

  • Open standards - prefer multi-vendor, interoperability-tested protocols.
  • Open-source - PyTorch/TensorFlow behave the same across vendor toolsets, so models stay portable.
  • Abstraction layers - decouple hardware from the operating platform; Terraform / Infrastructure-as-Code hides provider-specific config so you can swap vendors within a fabric without re-tooling.
  • In-house expertise - cross-train teams (RDMA/RoCEv2 knowledge is valuable on any stack).
  • Multi-vendor redundancy - active-active or active-standby across providers, with continuous cost/performance monitoring so no single vendor captures you.

Rack-level integration vs independent components

AI plant can be bought as integrated rack-scale systems - compute, fabric, power, and cooling co-engineered as one unit - or assembled from independent, loosely coupled components. Integration buys density, efficiency, and speed to deploy (the co-design is exactly what extreme kW/rack demands, Sustainability - Green AI, power and cooling), but it makes the rack the vendor's unit: refresh cadence, spares, and pricing follow the vendor, concentrating lock-in at rack granularity. Loose coupling preserves optionality - multi-vendor sourcing and per-subsystem incremental refresh - at some cost in engineering effort and peak density; done well it captures most of the benefit while keeping the exits open. Treat this as a lock-in decision with a physics constraint: the higher the density target, the stronger integration's pull.

Recommendation / justification

Treat interoperability as a first-class requirement when the business may change providers, burst to cloud, or run multi-cloud. Build on open standards + IaC abstraction + portable (open-source) frameworks, and design the network (VXLAN EVPN / SD-WAN) so workloads can move without re-addressing or re-engineering. Justify by business agility, cost control, DR, and compliance - the same drivers behind the cloud/hybrid decision in Service placement - on-prem, cloud, hybrid, distributed and Cost, ROI, and scalability of AI infrastructure.

What would change this (mid-scenario twist)

  • Hard data-sovereignty rule -> mobility is constrained to permitted regions; design placement around it.
  • Latency-critical training -> it does not move; only inference/data is mobile.
  • Heavy InfiniBand investment already in place -> weigh migration cost vs lock-in; UEC/Ethernet may be the exit path.

IPv6 / dual-stack note

Mobility across sites/clouds must preserve dual-stack reachability; ensure the overlay (VXLAN EVPN) and any SD-WAN policy carry IPv4 and IPv6 identically so a moved workload keeps both.

Spaced repetition

In AI workload mobility, what are the three things that can move, and how do training vs inference place?

The model, the inference application, or the data. Training stays central (resource-intensive, latency-coupled); inference is a candidate for the edge, moving toward user demand.

Which network technologies enable AI workload mobility across sites?

VXLAN EVPN (software-defined L2/L3 reachability that follows the workload) and SD-WAN (secure, prioritised flows across geographies).

Why is NVIDIA InfiniBand described as a vendor lock-in risk?

Its dominance in AI/HPC interconnect makes it a de-facto proprietary standard; Ethernet/UEC is the open alternative that mitigates the lock-in.

The four ways to mitigate AI vendor lock-in?

Open standards, open-source frameworks (portable PyTorch/TensorFlow), abstraction layers (Terraform/IaC), and in-house cross-trained expertise - plus multi-vendor active-active/standby redundancy.

Where do vendor dependencies typically hide?

Technology/integration, data and API formats, human capital (skills tied to one stack), and long-term contracts.

Integrated rack-scale AI systems vs independent components - what is the tradeoff?

Integration co-engineers compute/fabric/power/cooling for density, efficiency, and deployment speed but concentrates lock-in at rack granularity (vendor's refresh, spares, pricing); loose coupling preserves multi-vendor optionality and incremental refresh at some density/engineering cost - and the higher the kW/rack target, the stronger integration's pull.

Sources

  • Cisco U. learning paths: AI Workloads Placement and Interoperability (Workload Mobility, Vendor Lock-In Risks/Mitigation) and Architecture Migration to AI/ML Network.
  • Cisco Live BRKGRN-2419, Sustainability vs. Infrastructure Requirements - rack-level integration vs loose coupling.

domain: AI Infrastructure · type: design-note · status: complete · tags: · source: Cisco U. - AI Workloads Placement and Interoperability; Architecture Migration to AI/ML Network

Read more