Storage and checkpointing for AI

Requirement / business driver

Feed training data and absorb periodic checkpoints without starving the GPUs or stalling on I/O.

Two I/O patterns

• Training-data ingestion - sustained, parallel high read throughput to keep GPUs fed.
• Checkpointing - periodic, bursty, all-nodes-write-at-once (incast!) saves of model state for fault tolerance. If the checkpoint write is slow, every GPU idles until it finishes.

Storage protocols

GPUDirect Storage and software-defined storage

GPUDirect Storage opens a direct path from NVMe / NVMe-oF straight into GPU memory, with a DMA engine near the NIC/storage doing the move - bypassing the CPU and its bounce buffer. It matters most when datasets no longer fit in host memory and I/O becomes the bottleneck, and it pairs with GPUDirect RDMA (RDMA, RoCE and RoCEv2) to keep the CPU off the data path end-to-end. Software-defined storage (SDS) decouples storage management from the hardware, pooling any disks into a unified, scale-out resource; for AI the SDS types that fit best are scale-out file systems and object storage (the parallel-FS row above), with HCI suited to edge/on-prem. One selection criterion the parallel-FS row hides: classic parallel file systems buy their throughput with a proprietary client agent on every GPU node - a fleet-wide software, stability, and security surface - which is exactly what disaggregated NAS platforms target by delivering parallel-class performance over standard pNFS and S3 with no custom client.

The HCI data plane: replication factor and what to switch off

When the storage is the cluster (the hyperconverged archetype in AI-enabling hardware - GPU, DPU, SmartNIC), protection changes vocabulary: instead of array RAID or erasure stripes, the distributed store keeps RF (replication factor) copies of every write across nodes. The validated AI design runs RF2 - usable capacity is roughly half of raw before data reduction - with rebuild-capacity reservation enabled so a failed node's data can re-protect without filling the cluster. The data-reduction stance is deliberately asymmetric: inline compression on, deduplication off, erasure coding off - model weights, embeddings, and vector segments carry little duplicate data to find, and erasure coding buys capacity back at the cost of a heavier write path the serving tier should not pay. Embedded file and object services are dual-homed by design: the back-end leg sits in the same subnet as the controller VMs for data locality, the client leg on a separate subnet for exposure control. The DR posture completes the picture and pairs with a declarative stack: with platform and applications re-derivable from the Git repository (GitOps) and the vector store rebuildable from the source buckets, the backup set collapses to the data buckets plus the repo - replicate those bucket-level to an external S3 target and the standalone cluster needs no second site.

Checkpointing design

Tune frequency (fault-tolerance vs overhead), provide bandwidth to absorb the burst (incast -> buffers/lossless, Buffering, microbursts, and congestion signaling), and consider local NVMe + async flush (checkpoint to a fast tier, then drain to bulk storage). The storage network is usually a separate plane from the GPU back-end (Connectivity models and SDN in AI fabrics).

Dataset versioning: storage as the governance mechanism

Governance asks for lineage, versioning, and dataset-to-model traceability (Regulations, governance, and AI security policy); the storage array is where the mechanism actually lives. Array snapshots version a training dataset at a point in time - a read-only, space-efficient record that ties a model to the exact data it saw - and writable clones (NetApp FlexClone is the validated pattern) hand data scientists instant copies to reshape while the gold-source dataset stays immutable. Because clones consume capacity only for changed blocks, versioning stops being a cost decision and becomes a default. In regulated scenarios, treat snapshot-based dataset versioning as the storage layer's answer to an auditability requirement, not an optional convenience.

Storage strategy: tiering and caching

Beyond protocol choice, an AI storage strategy is usually multi-tier, matching media cost to access pattern. A hot tier of local NVMe/SSD serves active training data and checkpoints at lowest latency; a warm tier (networked scale-out or cloud) absorbs growing datasets and elastic, geo-distributed demand; a cold/archival tier holds infrequently accessed data cheaply for retention and compliance. Data tiering migrates data between tiers by access frequency and caching keeps the hottest data on the fastest media - together the main lever for balancing speed against capacity and cost (Cost, ROI, and scalability of AI infrastructure). Where the warm/cold tiers physically sit is also a placement and sovereignty decision (Service placement - on-prem, cloud, hybrid, distributed, Data sovereignty and data gravity).

One array-native mechanism deserves its name here: a FlexGroup-class volume presents a single parallelised namespace striped across controllers, aggregates, and member volumes with automatic load balancing - the scale-out-NAS answer to a training corpus of millions of small files hammered by parallel clients, and the NAS-side sibling of the object-store small-files fix below.

When one array serves production and development side by side, the sharing contract is storage QoS: per-volume ceilings cap the noisy neighbour, floors guarantee the training pipeline its IOPS regardless of what dev does, and adaptive QoS scales the guarantee with volume size so the policy survives growth - noisy-neighbour containment is a storage-policy decision before it is a fabric one.

The data-lake tier: from co-located Hadoop to disaggregated Ozone

The data estate feeding training and RAG has its own validated architecture. Classic Hadoop's founding premise was data locality - compute scheduled onto the nodes holding the blocks, storage and compute scaling as one. The validated evolution disaggregates: a stateless, containerised compute farm reaches the lake over the fabric, and the lake's storage layer becomes Apache Ozone. The move is forced partly by HDFS's own ceilings - Cloudera's supported wall is no more than ~100 TB per data node and no drives past 8 TB under HDFS - where Ozone scales to billions of objects, small and large files alike, by grouping blocks into storage containers (the management grain) and splitting namespace from storage (an Ozone Manager owns names, a Storage Container Manager owns containers - so the metadata half scales separately). Its second trick is a dual personality: HDFS-compatible and S3 APIs on one store - the S3 face is what lets disaggregated compute treat the lake as an object store, while data locality stays available to applications that still want it. Validated rules ride along: dedicated NVMe metadata drives on every Ozone node, and if erasure coding rs(6,3) is intended, the stripe width sets the floor - at least nine data nodes before the scheme even fits, so the protection choice, not the workload, dictates minimum cluster size. Node classes then split by economics: NVMe-dense performance nodes for the hot lake, 16 TB LFF capacity nodes as the low-$/TB tier the tiering strategy above retires data into (the platform shape this tier anchors is in AI-enabling hardware - GPU, DPU, SmartNIC).

RAG-serving state: the vector database's split profile

Inference-side AI brings its own storage tenant: the vector database behind RAG retrieval. The validated pattern (Milvus on the FlashStack RAG design) splits its persistence in two - hot cluster state on block persistent volumes (container CSI over dual-fabric iSCSI to all-flash block storage), while large-scale index files and binary logs go to S3-compatible object storage on a scale-out file/object platform. One application, two storage classes, by design. The consequence lands in the worker's network layout: beside the management vNIC and the two per-fabric iSCSI vNICs, the validated worker carries a fourth, dedicated object-storage interface on its own VLAN, so vector-index traffic neither rides the management path nor contends with block I/O (Storage networking - FC, FCoE, iSCSI, and NVMe-oF). When a scenario adds RAG, the storage design question is not "file or object?" but "block for the database's hot state, object for its bulk segments - and a network path for each" (AI-ML workloads and their infrastructure impact).

Who provisions the volume: CSI direct or through the hypervisor

Container persistent storage reaches the array by one of two validated paths. CSI direct (the Trident-class pattern this note's vector-database block already uses): the CSI driver talks to the array itself, and each persistent volume is an array-native volume reached over the worker's iSCSI / NVMe-TCP / NFS lanes. CSI through the hypervisor (the virtualised-OCP pattern): the driver talks to vCenter, never to the array - a claim becomes a first-class virtual disk on a VMFS datastore or a 1:1 vVol, and the array participates only by publishing its capability tiers (IOPS / latency classes) upward through VASA, where they surface as SPBM policies that Kubernetes StorageClasses name. The tiering contract becomes a policy chain - StorageClass -> storage policy -> datastore -> array volume - and the SAN underneath stays invisible to Kubernetes. The mediated path carries its own fine print: the vSphere driver supports expansion and snapshots but not volume cloning, vVols keep per-volume visibility and QoS where VMFS pools many volumes behind one datastore, and storage vMotion is off-limits under CSI volumes (the virtualisation fork in AI-enabling hardware - GPU, DPU, SmartNIC).

A third path sits between those two: the in-cluster SDS overlay (Portworx-class). Storage nodes run inside Kubernetes on every worker, carve their raw disks from hypervisor datastores (via the virtualisation manager's API) or arrays, aggregate them into their own storage cluster, and replicate at the container-volume level - buying container-granular snapshots, encryption, RBAC, and disaster recovery (near-zero-RPO metro plus async) at the cost of operating a second storage system above the array. Two interactions matter for design. Capacity: overlay replicas that all land on one deduplicating array multiply the array's data-reduction rate by the replica count - the copies dedup against each other, so the overlay's protection is nearly free in array capacity. Mobility: the overlay bans moving its backing virtual disks between datastores - the same no-storage-migration rule as the plain hypervisor CSI path, now confirmed a third time. And when the overlay's direct-access mode provisions array volumes 1:1 onto claims, the path collapses back toward CSI-direct - the fork is a spectrum, chosen by how many storage services the platform team wants to own.

Recommendation / justification

Use a parallel / software-defined file system (often over NVMe-oF/RoCE) for performance, object storage for the data lake, and design checkpointing for the incast burst (fast tier + enough fabric bandwidth). Justify by the two patterns: sustained read throughput for ingestion, burst-absorbing write for checkpoints - both of which can stall GPUs if undersized.

What would change this (mid-scenario twist)

• Checkpoint writes stall training -> faster tier (local NVMe), more storage-fabric bandwidth, lossless.
• Huge unstructured datasets -> object/S3 data lake plus a parallel FS cache.

Validation checks

• Can the storage path sustain GPU read demand and absorb the checkpoint incast?
• Is the storage plane sized/isolated so it doesn't contend with the GPU back-end?

IPv6 / dual-stack note

IP-based storage (NFS/object) and NVMe-oF/TCP run over v4 or v6; keep MTU/jumbo consistent (MTU, fragmentation, and PMTUD).

Related

• Latency and the straggler problem
• Bandwidth and capacity planning
• Buffering, microbursts, and congestion signaling
• Connectivity models and SDN in AI fabrics
• Storage networking - FC, FCoE, iSCSI, and NVMe-oF

Spaced repetition

Sources

• Cisco/NetApp/SUSE, FlexPod with SUSE Rancher for AI design guide (Dec 2023) - FlexGroup parallelised-namespace mechanism; min/max/adaptive storage-QoS sharing contract; full 75-page pass 2026-07-26 (user-supplied PDF).
• Cisco, CDIP with Cloudera Data Platform Private Cloud for GenAI (CVD, May 2024) - Ozone architecture, HDFS ceilings, EC rs(6,3) floor, NVMe metadata rule, node classes; design chapters passed 2026-07-25.
• Cisco/Pure, FlashStack for AI: MLOps CVD (Feb 2024) - Portworx-class in-cluster SDS overlay, replica-vs-dedup interaction, direct-access collapse, datastore-mobility ban (third confirmation); passed 2026-07-25.
• Cisco/Hitachi, Adaptive Solutions Red Hat OCP AI CVD (Oct 2024) - vSphere CSI/CNS, VASA/SPBM policy chain, vVols/VMFS, no-clone and no-storage-vMotion constraints; captured through the Implementation chapter 2026-07-25.
• Cisco, Compute Hyperconverged with Nutanix GPT-in-a-Box 1.0/2.0 CVDs (2024/2025) - RF2 stance, data-reduction settings, dual-homed Files/Objects, GitOps-collapsed backup set; design chapters passed 2026-07-25.
• Cisco AI POD for Enterprise Training and Fine-Tuning Design Guide (CVD, Jan 2026) - standard-protocol pNFS/S3 vs proprietary parallel-FS clients
• Cisco, Cisco AI PODs with FlexPod Datacenter solution overview (Feb 2026) - NFSv4.1-over-RDMA/FlexGroup/S3-artifact storage pattern; fetched in full 2026-07-25.
• Cisco, Scaling FlexPod for GPU Intensive Applications CVD (Dec 2023) - FlexClone/snapshot dataset-versioning pattern; passed through the HPC-X setup section 2026-07-25 (SpecHPC test-result chapters not captured).
• Cisco, FlashStack for Enterprise RAG Pipeline with NVIDIA NIM CVD (May 2025) - Milvus block+object split, worker vNIC layout; passed through the Solution Design chapter 2026-07-25 (vector-DB benchmark chapters not captured).
• Cisco AI/ML blueprint; parallel-filesystem (Lustre/GPFS/WEKA/VAST) and NVMe-oF references.
• Cisco U., "Data Storage Strategy" (Storage Resources) - tiering, caching, on-prem/cloud/archival mix.

domain: AI-Infra · blueprint-ref: AI-Infra 4.4 Storage (strategy, FC/FCoE/NVMe-oF, IP-based) · type: design-decision · status: complete · tags: [elective/ai-infra, ai/storage, tradeoff/latency, tradeoff/scale]