AI: AI-ML fundamentals & governance

Service placement - on-prem, cloud, hybrid, distributed

Miro Kosut

Miro Kosut

— 3 min read
Share

Requirement / business driver

Decide where AI workloads run, balancing data gravity, cost/ROI, GPU availability, latency, and compliance.

Functional vs non-functional requirements

Placement decisions hang on two requirement types. Functional requirements define what the system does - scalability, integration with existing infrastructure/services, user experience, security capability. Non-functional requirements define how it performs - latency, cost, reliability, availability, regulatory compliance, geographic distribution. The placement that wins is the one that best satisfies whichever non-functional requirement dominates: latency/control favour on-prem, cost/elasticity favour cloud, a mix favours distributed/hybrid.

Options

  • On-premises - full control, data stays put, but CapEx and you own power/cooling.
  • Public cloud - elastic OpEx, fast start, GPU capacity if available.
  • Hybrid - keep sensitive data/training on-prem, burst to cloud.
  • Distributed / edge - inference near users or data sources for low latency.

Comparison

ai-placement-matrix

Drivers

Data gravity (move compute to the data - Data sovereignty and data gravity), cost/ROI (Cost, ROI, and scalability of AI infrastructure), GPU scarcity, latency to users, and compliance/sovereignty.

Model availability constrains placement

The model's weight availability decides which placements are even eligible, before any infrastructure comparison. Closed-weight frontier models are consumable only as a hosted API - prompts, retrieved context, and outputs transit the provider's boundary - which is incompatible with air-gapped hosting and hard to square with strict sovereignty (AI security design and hosting models - air-gapped, private, public). Open-weight models can be self-hosted, making them the only eligible class for air-gapped or sovereignty-bound designs - at the price of owning the serving stack (GPU inference capacity, scaling, patching) that a hosted API would have absorbed. Screen the model class against the placement constraint first; then optimize within the eligible class.

Recommendation / justification

Training is usually centralized (on-prem or dedicated cloud region) where the GPUs, power, and data gravity are; inference is often distributed/edge or cloud near users; hybrid keeps regulated data local while bursting elastic demand to cloud. Connect via the WAN/cloud designs in SD-WAN design / WAN transport architecture selection. Justify by the dominant constraint (data gravity/compliance vs elasticity/latency).

What would change this (mid-scenario twist)

  • Strict data residency -> on-prem / in-region (sovereignty wins over elasticity).
  • Spiky/variable demand or no on-prem GPUs -> cloud or hybrid burst.
  • Real-time user latency -> distributed/edge inference.

Validation checks

  • Does placement respect data gravity and compliance first, then cost/elasticity?
  • Is the cloud/WAN connectivity sized for the data movement implied?

IPv6 / dual-stack note

Cloud and WAN connectivity should be dual-stack; verify v6 reachability to cloud AI services.

Spaced repetition

Where do training vs inference typically sit?

Training centralized (on-prem or dedicated cloud, where GPUs/power/data gravity are); inference distributed/edge or cloud near users.

The first questions in AI placement are [...] - then cost, elasticity, and latency.

The first questions in AI placement are data gravity and compliance/sovereignty - then cost, elasticity, and latency.

Hybrid placement is chosen to [...].

Hybrid placement is chosen to keep sensitive/regulated data on-prem while bursting elastic demand to the cloud.

Functional vs non-functional requirements in a placement decision?

Functional = WHAT the system does (scalability, integration, UX, security capability); non-functional = HOW it performs (latency, cost, reliability, availability, compliance, geo). The dominant non-functional requirement usually drives on-prem vs cloud vs distributed.

Why does open- vs closed-weight model availability gate placement decisions?

Closed-weight models exist only behind a hosted API (data transits the provider - incompatible with air-gap, hard for strict sovereignty); open-weight models can be self-hosted - the only eligible class for air-gapped/sovereign designs, at the cost of owning the GPU serving stack the API would have absorbed.

Sources

  • Cisco Live DEVNET-3707 - Network Telemetry and AI for Network Incident Response (model-availability taxonomy)
  • Cisco AI/ML blueprint; cloud GPU and hybrid-AI references.

domain: AI-Infra · blueprint-ref: AI-Infra 1.2 Service placement (on-prem, cloud, distributed) · type: design-decision · status: complete · tags: [elective/ai-infra, ai/fundamentals, tradeoff/cost, tradeoff/operability]

Read more