AI-assisted operations - AIOps
In one line
AIOps applies ML to the flood of telemetry and logs an AI fabric produces - to detect anomalies, correlate events, and (eventually) auto-remediate - because thousands of GPUs and sub-second telemetry are beyond human monitoring.
The three blueprint pieces
- Data scrubbing (4.3.a) - clean, normalize, and enrich raw telemetry/logs (dedup, unify formats, filter noise, fill gaps) so analytics are trustworthy. Garbage in, garbage out - the scrubbing pipeline is the unglamorous foundation.
- Log correlation (4.3.b) - tie together events across thousands of devices/GPUs and across time - time-aligned by PTP (PTP timing for AI fabrics) - to find root cause (e.g., correlate a job slowdown to one flapping optic).
- Telemetry analysis (4.3.c) - ML over streaming telemetry to spot anomalies, predict failures, find the straggler (Latency and the straggler problem), and recognise congestion patterns.
Why it matters in AI fabrics
The scale (thousands of GPUs/devices, sub-second telemetry) makes manual monitoring impossible, and finding one slow link among thousands demands automated correlation. AIOps is the operational counterpart to the performance obsession of this elective - it's how you protect JCT in production.
AIOps use cases (beyond the fabric)
The same telemetry-plus-ML pattern powers a broader set of network use cases: predictive analytics (anticipate failures from trends), network optimisation (tune config/paths), self-healing automation (auto-remediate known faults), capacity planning and forecasting (size ahead of demand), predictive risk management, and faster incident response. All rest on the same foundation - rich, time-aligned telemetry feeding ML correlation - and all keep a human in the loop where blast radius is high.
Closing the loop
AIOps feeds closed-loop / event-driven automation (CI-CD pipelines for networks (NetDevOps) and Core closed-loop design): sense (telemetry) -> ML detect/correlate -> decide -> act (gated). It is built on PTP-aligned, model-driven streaming telemetry (Network management - traditional vs model-driven).
Tuning the lossless fabric by telemetry (Nexus Dashboard Insights)
Lossless behaviour is set by thresholds, then verified by telemetry - you cannot eyeball it. Nexus Dashboard Insights consumes the switches' hardware flow telemetry (flow table + flow-table events, per-device / per-interface / per-flow), exposing ECN-mark counters and PFC RX/TX pause counters per class-of-service. Use it as a closed tuning loop:
- Stop the drops first - in normal traffic, tune WRED/AFD (Lossless fabric - PFC, ECN, DCQCN) thresholds until drops cease (the single most important step).
- Handle the bursts - for incast/microburst, tune WRED/AFD together with PFC until behaviour is fully lossless.
- Then optimise - use the ECN-mark and PFC RX/TX counters to push for highest performance once loss is gone.
The same intent-driven tooling also flags configuration drift (state vs operator intent), the operational complement to building the fabric from templates (Validated reference design - Cisco AI-ML lossless fabric (CVD)).
GenAI in the loop - filtering, semantics, and agentic diagnosis
Using an LLM as the analysis engine imposes a design constraint of its own: the data volume dwarfs the model's attention budget. A single moderate log file runs to hundreds of thousands of tokens, and one device's telemetry snapshot can expose thousands of features - so a relevance-ranking stage belongs in the scrubbing pipeline (4.3.a). Statistical and embedding methods do the shrinking: e.g. TF-IDF-sort a log by rarity so the rare, informative lines surface and the repetitive all-is-well noise sinks - then hand only the top slice to the model (or the human).
For streaming telemetry (4.3.c) the humbling finding is that change is not a signal: on a real device the large majority (~70%) of features change constantly, so unsupervised change-point detection becomes diagnostic only when filtered by the type of change (step, spike, variance shift) and by feature importance inferred from the feature's own name - name tokens that are rare across the feature set (an admin-down interface count) matter more than ubiquitous ones (per-interface byte counters). That semantic filter turns a wall of change points into a shortlist a person can act on.
For cross-domain incident diagnosis, the emerging architecture mirrors the human loop - hypothesize -> retrieve -> evaluate, then analyze and summarize - as cooperating agents: one classifies the issue and generates cause hypotheses, one retrieves from knowledge bases / observability systems / the live network, one verifies-or-disproves and maps causal dependencies, and one integrates the root-cause verdict with a confidence score. The design ingredients: context (topology/inventory, knowledge bases, a human in the loop), the full CMELT data set (config, metrics, events, logs, traces), team memory of past investigations, purpose-fit models, and explicit termination decisions. Cost reality: one investigation is thousands of small-model calls over minutes - this augments L1 triage; it is not a line-rate control loop. And name the philosophical fork for the scenario: deductive expert systems (fit data to a human-defined schema - closed world, deterministic, customized per deployment, ontologies to maintain) vs inductive GenAI diagnostics (derive the schema from the data - open world, generically applicable, probabilistic). Pick deductive where determinism and auditability rule; inductive where breadth and unstructured data dominate.
Containing the nondeterminism - workflow vs agent, and the actuator boundary
Once GenAI graduates from analysis to incident response, the controlling design axis is how much control flow the model owns. At one end sit workflows - the LLM is embedded in predefined code paths (chained or parallel steps), or at most directs flow through predefined paths (routing to a specialist, an orchestrator fanning work to workers, an evaluator looping an optimizer). At the other end sits the autonomous agent - the LLM directs its own actions from environmental feedback. Predictability, auditability, and cost-boundedness all fall as autonomy rises, so device-touching operations belong at the workflow end: a fixed graph (coordinate -> orchestrate -> gather -> analyze, with a bounded revisit loop -> report) in which the model reasons inside the nodes while deterministic code owns the edges.
The complementary rule is the actuator boundary: because LLM output is stochastic, the model itself never touches a device - every read and every action is mediated by a scoped, deterministic automation-API layer (an automation server fronting the estate), which becomes the single point of authorization, audit, and gating (Telemetry-driven and closed-loop automation). The tool contract is the safety boundary - typed signatures, precise docstrings, deliberately narrow scope - and each agent's prompt is pinned with an explicit negative scope ("identify and describe - do not investigate or remediate"). Feed both directions through structured interfaces (Network management - traditional vs model-driven), and let the same graph accept two entries - a telemetry-alert webhook and a human request - so machine-triggered and operator-triggered response share one audited path.
Recommendation / justification
Build it in order: (1) rich streaming telemetry + PTP time alignment + a scrubbing pipeline (foundation), (2) ML correlation / anomaly detection, (3) selective, gated auto-remediation (human-in-the-loop for high blast radius). Justify by scale (un-monitorable by hand) and JCT protection (catch stragglers and failing optics fast).
What would change this (mid-scenario twist)
- Massive fleet, slow manual triage -> prioritise AIOps correlation + anomaly detection.
- High-risk actions -> keep remediation gated/assisted, not fully autonomous.
Validation checks
- Is telemetry time-aligned (PTP) and scrubbed before analysis?
- Does AIOps actually close a loop (detect -> correlate -> act), with high-risk actions gated?
IPv6 / dual-stack note
Telemetry/log pipelines are address-agnostic; ensure collection and correlation cover both v4 and v6 paths.
Related
- Latency and the straggler problem
- PTP timing for AI fabrics
- Network management - traditional vs model-driven
- CI-CD pipelines for networks (NetDevOps)
Spaced repetition
What are the three AIOps pieces in the blueprint?
Data scrubbing (clean/normalize telemetry), log correlation (tie events across devices/time, PTP-aligned), and telemetry analysis (ML anomaly/straggler/failure detection).
Why is AIOps necessary for AI fabrics specifically?
Thousands of GPUs/devices with sub-second telemetry are beyond human monitoring - finding one slow link/straggler among thousands needs automated correlation (protecting JCT in production).
The right build order for AIOps is [...].
The right build order for AIOps is telemetry + PTP alignment + scrubbing (foundation) -> ML correlation/anomaly detection -> gated auto-remediation.
What is the telemetry-driven order for tuning a RoCEv2 lossless fabric?
(1) tune WRED/AFD until drops stop in normal traffic, (2) tune WRED/AFD + PFC for incast/microburst until fully lossless, (3) use ECN-mark and PFC RX/TX counters to optimise - all verified per-flow via Nexus Dashboard Insights.
Name several AIOps use cases beyond congestion tuning.
Predictive analytics (failure anticipation), network optimisation, self-healing automation, capacity planning/forecasting, predictive risk management, and faster incident response - all built on telemetry + ML correlation.
Why does an LLM-based AIOps pipeline need a relevance-filtering stage, and name a simple mechanism.
Logs/telemetry exceed any model's context budget (one log file can be hundreds of thousands of tokens), so scrubbing must rank and shrink - e.g. TF-IDF-sorting log lines by rarity so rare informative lines surface and repetitive noise sinks.
Why is raw change detection insufficient for telemetry analysis, and which two filters fix it?
~70% of a device's features change constantly, so change alone is noise - filter change points by type of change (step/spike/variance) and by feature importance inferred from name-token rarity (rare tokens ~ high diagnostic value).
Deductive vs inductive network diagnostics - the design contrast?
Deductive expert systems fit data to a human-defined schema (closed world, deterministic, customized, ontology upkeep); inductive GenAI derives the schema from data (open world, generically applicable, probabilistic) - choose by whether auditability/determinism or breadth/unstructured data dominates.
Agentic AIOps diagnosis grounds its hypothesize -> retrieve -> evaluate loop in [...] data - config, metrics, events, logs, traces - plus topology context and team memory of past investigations.
Agentic AIOps diagnosis grounds its hypothesize -> retrieve -> evaluate loop in CMELT data - config, metrics, events, logs, traces - plus topology context and team memory of past investigations.
What is the workflow-vs-agent autonomy spectrum, and where should device-touching GenAI operations sit?
Workflows embed the LLM in predefined code paths (fixed chains, or LLM-directed routing / orchestrator-worker / evaluator-optimizer through predefined paths); agents let the LLM direct its own actions from environmental feedback. Predictability, auditability, and cost-boundedness fall as autonomy rises -> device-touching operations sit at the workflow end: the model reasons inside nodes, deterministic code owns the edges.
Because LLM output is stochastic, the actuator-boundary rule says [...] that is the single authorization, audit, and gating point.
Because LLM output is stochastic, the actuator-boundary rule says the model never touches a device - every read and action is mediated by a scoped deterministic automation-API layer that is the single authorization, audit, and gating point.
Sources
- Cisco Live DEVNET-3707 - Network Telemetry and AI for Network Incident Response (workflow-vs-agent spectrum, actuator boundary)
- Cisco AI/ML blueprint; AIOps / streaming-telemetry and closed-loop automation references.