PTP timing for AI fabrics

In one line

Precise time sync (PTP / IEEE 1588) gives the fabric a common, sub-microsecond clock - essential for correlating telemetry, measuring congestion, and coordinating distributed workloads.

PTP basics

Hardware timestamping, a grandmaster clock, and boundary / transparent clocks in the path deliver sub-microsecond accuracy - versus NTP's millisecond range. A PTP profile defines the parameters; holdover covers grandmaster loss.

What buys the three-orders-of-magnitude jump over NTP is where the timestamp is taken: PTP's event messages (Sync, Delay_Req and the peer-delay pair) are stamped in hardware at the PHY, below the OS, driver, and queueing jitter that caps a software-timestamped protocol at milliseconds. In the common two-step mode the sender emits Sync first and delivers the precise departure time in a Follow_Up general message, so the forwarding path never waits on the stamping path; offset and path delay then fall out of the four-timestamp delay request-response exchange (t1-t4) between parent and member. The alternative to distributing time in-band is a GNSS/GPS (or IRIG) receiver per node - about 100 ns, but a separate, non-Ethernet cabling plant at every site, which is exactly the cost-and-scale problem PTP exists to remove: one disciplined grandmaster, distributed over the network you already have.

Clock roles, BMCA, and grandmaster pinning

Four roles make up the hierarchy: the grandmaster (primary reference), ordinary clocks (single-port endpoints acting as parent or member), and the two in-path roles below. The hierarchy is not configured - it is elected. The BMCA (Best Master Clock Algorithm) runs on every port, every announce interval, comparing the local dataset against received Announces in strict order: Priority1 -> clockClass -> clockAccuracy -> variance -> Priority2 -> clockIdentity, with hop count to the master as the tie-breaker. Port states (listening, master, slave, passive) fall out of the election, and passive is the timing plane's own loop prevention: in redundant topologies the redundant timing path is parked passive, so PTP never loops even where the data plane meshes. The design lever hides in the ordering - everything after Priority1 describes intrinsic clock quality, so Priority1 is the one administrative knob: set it to pin the primary GM and stage the backup GM, and failover needs no protocol help - lose the better Announce for long enough and the next-best clock simply wins the next election.

Boundary vs transparent clocks - and PTP-unaware transit

The two in-path roles solve different problems. A boundary clock terminates the flow: it recovers time with its own servo and regenerates fresh PTP on every downstream port - which collapses the grandmaster's fan-out (the GM serves a handful of BCs instead of thousands of endpoints), contains message domains, and crosses L2/L3 boundaries because each port speaks new PTP. The cost is that every BC adds its servo's filtering error, so very long BC chains accumulate drift. A transparent clock never terminates: it measures its own residence time and adds it to the correction field of the passing message, so the end member still disciplines directly against the GM - near-zero added error, but the GM's message load is not reduced and the messages must be able to reach the member at all. Which brings in the two delay mechanisms: end-to-end delay request-response measures the whole parent-member path, so it tolerates PTP-unaware hops - but every microsecond of uncorrected queueing variance on those hops lands directly in the offset; peer-to-peer (Pdelay) measures each link using link-local multicast that bridges do not forward, so every hop must participate. The overlay corollary matters in this vault: an SDA/VXLAN fabric can relay PTP as flooded overlay multicast without joining the timing domain (Connectivity models and SDN in AI fabrics) - it works, but transit queueing is invisible and uncorrected; acceptable for coarse sync, wrong where the budget is tight, where the requirement becomes BC/TC on-path. And since on any non-participating hop the queueing of the timing packets is the error, the QoS design must schedule PTP in the low-latency/EF class - the AES67 profile mandates exactly that for clock traffic.

Profiles are non-interoperable islands

A profile pins transport, message rates, and the on-path-support model - and profiles, like PTP v1 vs v2, do not interoperate: one profile per timing domain, with deliberate boundaries where ecosystems meet. The default profile runs over IPv4/UDP multicast at L3 in a single domain. 802.1AS (gPTP) is L2-only, expects every bridge to participate, is mutually exclusive with the default profile on the same switch, and comes up implicitly with AVB. AES67 is the media-interoperability profile that lets Dante/Livewire/Ravenna/Q-SYS islands share one 1588-2008 clock and mandates the QoS treatment (clock = EF 46, media = AF41, marked by the endpoints per spec). G.8275.1 is the telecom phase/time profile behind 4G/5G radio alignment: full timing support from the network - a T-BC at every hop in a T-GM -> T-BC chain -> T-TSC - over L2 multicast, with its own reserved domain range (24-43). The question a profile answers is design-shaped: who is required to participate, at which layer, for which consumer ecosystem.

Why it matters in AI fabrics

• Telemetry correlation: aligned timestamps across thousands of devices/GPUs are what let you locate the straggler (Latency and the straggler problem) and reconstruct a congestion event - the foundation for AI-assisted operations - AIOps.
• Measurement: precise one-way latency/congestion analysis needs synchronised clocks.
• Workload coordination: some distributed pipelines and inference/streaming use cases need tight time alignment.

Deterministic Ethernet as contrast - the AVB/TSN triad

Campus media and OT show the other way to make Ethernet predictable. 802.1BA (AVB) bundles three planes that must all be present: gPTP for common time; MSRP (802.1Qat) for per-stream admission control - the talker advertises, every hop checks resources, the listener's Ready completes the reservation, and bandwidth plus a worst-case latency bound are guaranteed end-to-end or the stream is refused; and 802.1Qav (FQTSS) credit-based shaping at every hop to deliver what was promised. The classes are engineered, not statistical: SR class A bounds roughly 2 ms (CoS 3), SR class B roughly 50 ms (CoS 2), reservations are capped at 75% of egress and granted first-come-first-served, with the remainder protected for everything else. The hard constraint is contiguity: every bridge on the path must be AVB-capable - a non-AVB switch turns the port into a boundary port and splits the domain. The contrast with the AI back-end is the useful part: PFC/ECN buys losslessness statistically, with no per-flow admission (Lossless fabric - PFC, ECN, DCQCN), and scales to elastic thousands of flows; MSRP buys hard per-stream guarantees, but only for modest, known stream counts. Few engineered streams with latency contracts -> reservation; massive elastic collectives -> statistical losslessness. One collision to name: SR class A rides CoS 3, the same lane FCoE claims for no-drop and UC uses for CS3 signaling (Storage networking - FC, FCoE, iSCSI, and NVMe-oF) - on converged trunks CoS 3 is now triple-booked.

Recommendation / justification

Deploy PTP with redundant grandmasters and boundary clocks, on switches/NICs that support hardware timestamping; pick a profile and plan holdover. Justify by the need for trustworthy, time-aligned telemetry at fabric scale (and any workload that mandates sync). Feed it into observability (Network management - traditional vs model-driven).

What would change this (mid-scenario twist)

• Heavy reliance on telemetry/AIOps -> PTP becomes mandatory, not optional.
• Grandmaster reliability concerns -> redundant GMs + boundary clocks + holdover.

Validation checks

• Do switches/NICs do hardware PTP, with redundant grandmasters?
• Is timing accurate enough to correlate fabric telemetry meaningfully?
• Stream the PTP oper data (offset-from-master, mean path delay, GM identity changes) as telemetry - a silent GM flap or a creeping offset should be graphed, not discovered.
• One profile and one domain plan per timing island, with explicit boundaries where versions/profiles/ecosystems meet?

IPv6 / dual-stack note

PTP operates independent of the data-plane family; ensure the timing distribution path exists in the management family. Concretely: platform PTP transport is commonly L2 or IPv4/UDP only (Catalyst is exactly that), so in a v6-first design either carry timing at L2 (gPTP) or verify v6 PTP transport per platform - never assume it.

Related

• Latency and the straggler problem
• Network management - traditional vs model-driven
• Lossless fabric - PFC, ECN, DCQCN - statistical losslessness vs MSRP's per-stream admission
• Connectivity models and SDN in AI fabrics - overlays relay PTP without joining the timing domain
• Storage networking - FC, FCoE, iSCSI, and NVMe-oF - the CoS 3 triple-booking

Spaced repetition

Sources

• IEEE 1588 PTP; Cisco AI/ML data-center timing guidance.
• Cisco Live BRKENS-2094 - Media & Time-Sensitive Networking with C9K (PTP mechanics, BMCA, BC/TC, profiles, AVB).

domain: AI-Infra · blueprint-ref: AI-Infra 4.5 Timing protocols (PTP) - applied to the fabric · type: design-decision · status: complete · tags: [elective/ai-infra, ai/network, tradeoff/operability]