AI-ML workloads and their infrastructure impact

In one line

Different AI workloads - and above all training vs inference - stress infrastructure in very different ways, and that profile is what drives every fabric, compute, and storage decision in this elective.

The workload spectrum

Classic ML (often CPU, lighter) -> deep learning (GPU) -> LLM / GenAI (massive GPU clusters, huge models and datasets). As you move up the spectrum, compute, network, and storage demands grow by orders of magnitude.

Training vs inference (the key split)

• Training - builds the model: massive sustained GPU compute over days/weeks, east-west lossless RDMA fabric for collective communication, huge datasets plus checkpoint incast, optimised for throughput / Job Completion Time.
• Inference - serves the model: lighter per request but very high volume, north-south / user-facing traffic, per-request latency sensitive, and elastically auto-scaled, often distributed / at the edge.

Comparison

Traditional (symbolic) vs modern (ML/DL) AI

Traditional / symbolic AI (rule-based "expert systems") encodes human knowledge as explicit rules and logic (a knowledge base + an inference engine, semantic networks, frames). It is interpretable but brittle and unscalable - every case must be programmed and it cannot generalise to novel input. Machine learning instead learns patterns from data and adapts; deep learning adds neural networks for tasks (vision, language) symbolic AI can't do. The shift from rules to data-driven learning is why modern AI exists - and why the infrastructure demands (GPUs, lossless fabrics) appear only with ML/DL, not with rule engines.

GenAI and LLM-specific considerations

Generative AI creates new content (text, images, code). One classic architecture is the GAN (Generative Adversarial Network) - a generator and a discriminator trained against each other. For language, the design choice that touches infrastructure is LLM vs SLM: Small Language Models are cheaper, lower-latency, and edge-deployable with a narrower scope, while large LLMs need the full training cluster. GenAI inference is memory-bound (huge parameter counts and KV-cache), so it rewards resource optimisation and, for SLMs, edge placement. RAG (Retrieval-Augmented Generation) grounds a model on a private knowledge base (a vector store + retrieval) and is often run on-prem for data privacy. Bias and fairness in generated output is a governance concern (Regulations, governance, and AI security policy).

RAG as an infrastructure workload

RAG deserves its own workload profile, because it is really two pipelines with opposite shapes. The ingestion path runs offline and throughput-shaped: documents are parsed, chunked, and pushed through a GPU-accelerated embedding model, and the resulting vectors are written to a vector database - batch work that can be scheduled like training. The serving path runs online and latency-shaped: the query is embedded, the vector database is searched, a reranking model re-orders the candidates, and only then does the LLM generate - so a database round-trip and two auxiliary models sit inside the user's latency budget. The validated anatomy (Cisco's FlashStack RAG design with NVIDIA NIM is the reference) therefore deploys three separately sized GPU services - LLM inference, text embedding, and reranking - not one, each scaled on its own, with the vector store's storage profile handled in Storage and checkpointing for AI. Design consequence: an "inference" requirement that turns out to be RAG brings a database, an object store, and two extra model services with it - budget them explicitly rather than sizing a lone LLM endpoint.

Inference SLAs: decomposing "low latency"

"Low latency" is not one number for an LLM endpoint - the validated metric set splits it into parts that stress different resources. Time-to-First-Token (TTFT) is the wait from prompt to first output token - the prefill phase, compute-bound, and what users perceive as responsiveness. Time per Output Token (TPOT) (inter-token latency) is the streaming rate after that - the decode phase, bound by GPU memory bandwidth as weights and KV-cache are re-read per token. End-to-end request latency ~= TTFT + TPOT x output tokens, and the system-side counterparts are requests/s and aggregate tokens/s - all shaped by concurrency, batch size, and input/output token lengths. The design consequence: write and validate the SLA per component, because batching raises aggregate throughput while stretching TTFT - the same hardware can pass the throughput target and fail the responsiveness one. Two boundary notes from the same design: production inferencing normally rides existing 100/200G front-end fabrics and an inference-only build needs no back-end fabric at all (Validated reference design - Cisco AI-ML lossless fabric (CVD)); and emerging agentic workloads - multi-step tool-and-model chains - multiply round-trips inside one user request, pushing sustained rates and latency sensitivity upward.

Two validated refinements sharpen the SLA picture. First, TPOT has a defensible numeric anchor: ~100 ms per output token - the reading speed of a fast human reader - is the industry-accepted ceiling for chat-style serving, and it is the bar Cisco's CPU-inference validation was tested against (AI-enabling hardware - GPU, DPU, SmartNIC). Second, not every inference tier is a session endpoint at all: the validated medical-imaging pattern runs inference as an event-driven, serverless function - an object-store bucket notification feeds a message topic and a scale-from-zero function processes each arriving image - so the metric becomes per-event completion and cold-start tolerance, not TTFT/TPOT. That shape autoscales with event volume, tolerates batch-like economics, and is exactly where CPU serving fits without any GPU pool.

The OpenVINO RAG validation adds the acceptance anchors and one lifecycle constraint. TTFT under ~2 seconds is the accepted responsiveness bar (the validated CPU pipeline ran at <=0.4 s), and the TPOT ceiling is really a band derived from reading speed: ~0.2 s/token from the 200-words-per-minute average reader down to ~0.1 s/token for a fast one - state which end the SLA assumes. And a RAG lifecycle rule: the serving embedding model must be the one that populated the vector store, so changing embedding models invalidates the store and forces full re-ingestion - retain the raw corpus so it can be rebuilt (Storage and checkpointing for AI).

Model memory footprint and GPU sizing

A separate first question is how much GPU memory the model needs, because that decides whether it runs on one GPU or must be sharded across several. A model's memory footprint is set by four things: the parameter count (more parameters -> more memory to hold weights), the numeric precision (FP32 uses twice the memory of FP16/BF16, and quantization to 8-bit or lower shrinks it further, trading some accuracy for footprint and speed), the batch size, and the sequence length (both inflate the activations and KV-cache held during a forward pass). The same model is therefore much cheaper to serve than to train: inference at reduced precision on a short context can fit in a fraction of the memory that full-precision training with long sequences demands. The design consequence is concrete - choose GPU VRAM so the target workload fits with headroom (a 7B model needs roughly 16 GB to serve but ~2x or more to fine-tune, so 48-80 GB cards leave room to run most models on a single GPU). When a model exceeds one GPU's memory it must be split across GPUs (model/tensor parallelism), which turns an intra-server problem into a scale-out fabric problem (Scale-up vs scale-out) - so memory sizing and fabric sizing are linked. Shrinking the model with quantization, pruning, or distillation (Sustainability - Green AI, power and cooling) is the lever that keeps a workload on fewer GPUs.

The second sizing lever is the method: customizing beats training by orders of magnitude, and the fine-tuning technique spans another order on its own. Training a 405B-class model from scratch consumed on the order of 16,000 H100s, while full fine-tuning of the same model fits in a few dozen current-generation GPUs; for a 70B model, walking the ladder 32-bit full fine-tune (~9 GPUs) -> 16-bit (~5) -> 16-bit PEFT/LoRA (~2) -> 8/4-bit QLoRA (~1) collapses the job to a single card. Because enterprises overwhelmingly customize pre-trained models rather than train them, the validated enterprise cluster quantum is 32-64 GPUs, sized to the largest concurrent fine-tuning/RAG workload - not to foundational training (Cost, ROI, and scalability of AI infrastructure).

Why it matters for design

Training's collective communication (all-reduce/all-gather, RDMA, RoCE and RoCEv2) is the east-west traffic that demands the lossless, rail-optimized, non-blocking fabric (Latency and the straggler problem, Bandwidth and capacity planning); inference shapes a different, distributed design. Recognising the workload (training vs inference, and LLM scale) tells you what infrastructure the scenario actually needs (Service placement - on-prem, cloud, hybrid, distributed).

IPv6 / dual-stack note

Workload type is independent of the IP stack; the fabric it implies is dual-stack-capable.

Related

• Parallelism strategies and collective traffic patterns - the traffic these workloads generate.
• Service placement - on-prem, cloud, hybrid, distributed
• Latency and the straggler problem
• Bandwidth and capacity planning
• AI-enabling hardware - GPU, DPU, SmartNIC

Spaced repetition

Sources

• Cisco, AI Inferencing on UCS X-Series for Intel OpenVINO Based RAG Deployments (CVD, Dec 2024) - TTFT/TPOT acceptance anchors, embedding-model freeze; full prose pass 2026-07-25.
• Cisco, Generative AI Inferencing with UCS X-Series M7 / 5th Gen Xeon white paper (Mar 2024) - ~100 ms/token anchor; full-text pass 2026-07-25.
• Cisco, AI Inferencing with Intel AMX for Medical Image Analysis (CVD, Nov 2024) - event-driven serverless inference shape; partial pass 2026-07-25.
• Cisco AI POD for Enterprise Training and Fine-Tuning Design Guide (CVD, Jan 2026) - inference SLA metrics; fine-tuning method ladder and cluster quantum
• Cisco, FlashStack for Enterprise RAG Pipeline with NVIDIA NIM, NIM Operator, and RAG Blueprint CVD (May 2025) - RAG pipeline anatomy; passed through the Solution Design chapter 2026-07-25 (benchmarking/sizing chapters not captured).
• Cisco AI/ML data-center blueprint; Nvidia training/inference references.

domain: AI-Infra · blueprint-ref: AI-Infra 1.1 Impacts on infrastructure for different AI/ML use-cases · type: concept · status: complete · tags: [elective/ai-infra, ai/fundamentals, tradeoff/scale]