QoS design - models and strategy

Requirement / business driver

Protect latency/jitter/loss-sensitive applications (voice, video, storage, trading) across constrained links, reconciling application requirements with infrastructure limits.

Models

• Best-effort - no guarantees.
• IntServ (RSVP) - per-flow reservations; doesn't scale.
• DiffServ - per-hop behaviors (PHB) keyed on DSCP classes; the scalable standard.

Strategy

Classify and mark at the edge (set the trust boundary at access), then queue/schedule in the core. Tools: classification/marking, queuing (LLQ for voice, CBWFQ), congestion avoidance (WRED), and policing/shaping at edges and to the SP. Pick the number of classes from the business need - a 4-class model for simple environments up to 8-/12-class where voice/video/data each need distinct treatment (more classes = more granularity but more operational complexity). Where that trust boundary sits is itself an environment decision: the campus user edge is untrusted, so access classifies and re-marks, while in the data centre the hosts and hypervisor vSwitch are administratively controlled - the boundary moves into the host edge and the design leans on source marking (exactly how RoCEv2 hosts set the DSCP in Lossless fabric - PFC, ECN, DCQCN).

Example DiffServ class model

Engineering the classes - sizing rules, targets, and admission control

The class model comes with numbers. The strict-priority (EF) queue obeys the 33% rule: keep the sum of all priority queues within about a third of the link, because a PQ is internally FIFO - real-time flows contend with each other inside it - and a link dominated by strict priority makes every non-real-time application's response time swing wildly, destroying the coexistence QoS exists to provide. Never enable WRED on the priority queue (real-time traffic gains nothing from early drop), and leave at least ~25% for the default best-effort class - thousands of unclassified applications live there. The requirements side is equally numeric: voice needs <=150 ms one-way, <=30 ms jitter, <=1% loss; interactive video ~<=200 ms; broadcast video ~<=400 ms with loss down to ~0.1% - these budgets, decomposed into propagation (fixed), serialization (link speed), and queuing (the designable term), are what the queue design must honor. And a priority queue cannot protect itself: QoS protects classes from each other, while admission control (CAC) protects a class from itself - most CAC is off-path counting at the application layer (call managers), which works for predictable flows and simple topologies, while on-path RSVP covers variable-rate flows and multipath cores; RFC 5865 closes the loop by marking admitted voice with its own DSCP so the network can distinguish admitted from unadmitted real-time traffic. Finally, class counts follow a 4 -> 8 -> 12 expansion path (RFC 4594 is the 12-class reference): expand by subdividing existing classes so markings and treatments stay consistent through the migration, and expect the hard part to be organizational agreement on who gets which class, not the mechanics.

Hierarchical QoS (HQoS)

At a sub-line-rate edge - a physical 1 GE handoff provisioned at, say, 50 Mbps - flat QoS never engages, because the physical link never sees congestion even while the provisioned rate is saturated. HQoS fixes this with a parent policy (shaper to the sub-rate) plus a child policy (CBWFQ/LLQ classes): the shaper creates backpressure at the provisioned rate so the child queuing actually takes effect. Essential for SP handoffs, sub-rate circuits, and per-VLAN/per-tenant shaping.

Policing vs shaping (admission control)

Both keep traffic within a class/SLA profile but differ on excess: policing drops or re-marks at the contracted rate (no delay); shaping buffers and delays excess to smooth output (adds latency). Rule: police real-time/voice (delay hurts more than loss) and shape elastic traffic (and at SP handoffs). WRED/AQM manages the tail of the queue - dropping early to head off TCP global synchronisation (Buffering, microbursts, and congestion signaling) - while queuing manages the front.

The re-mark half deserves its own mechanism, because it is how AF actually works. Policers come as a ladder - single-rate two-color (conform/violate), RFC 2697 single-rate three-color (adds a burst-excess band), RFC 2698 dual-rate three-color (CIR and PIR) - and their design use is policing as marking: conforming traffic keeps AFx1, excess is marked down to AFx2, violating to AFx3 - not dropped. DSCP-based WRED then completes the machine inside each AF queue by dropping AFx3 earliest and most aggressively, then AFx2, then AFx1 - so under no congestion the burst still flows, and under congestion the traffic that exceeded its contract is the first to pay. Police to mark down and let congestion decide; drop-on-police is for traffic that should never transit at all.

MPLS DiffServ: E-LSP/L-LSP and tunneling modes

Carrying DiffServ across an MPLS core raises two design choices. How the class is encoded: an E-LSP infers the per-hop behaviour from the label's 3-bit EXP / Traffic-Class field (up to 8 classes, no extra labels - the usual choice), while an L-LSP derives it from the label itself (one LSP per class). How marking is tunnelled at the edges - the uniform / pipe / short-pipe modes - decides whether the customer's DSCP survives the core and whether the core appears in traceroute: uniform keeps one DiffServ domain end to end (ingress copies DSCP to EXP, egress copies any core change back, TTL propagated so the core is visible); pipe preserves the customer DSCP and hides the core, with the egress PE queuing on the provider's marking; short-pipe also preserves DSCP and hides the core but the egress PE queues on the customer's original marking. The choice is an SLA/trust decision - a shared DiffServ domain versus a provider-controlled core.

Mapping the enterprise model into a provider's classes

An SP sells 4, 6, or 8 classes of service; an enterprise model rarely matches, so the CE must collapse classes - and the collapse has rules. Map all real-time (voice, and interactive video where the SLA warrants) into the SP's priority class and check its policing profile fits the sum. Never mix TCP and UDP applications in one SP class: under congestion TCP backs off and UDP does not, so the UDP applications quietly starve the TCP ones inside the shared class - keep them in separate SP classes even when their importance is equal. Put control and signaling with the SP's control class, not best effort. And because many SPs re-mark at their edge, agree the mapping contractually and restore the enterprise markings at the far CE so the internal 8/12-class scheme survives the transit (MPLS L3VPN and L2VPN).

QoS in virtualized networks

Virtualizing the network multiplies logical networks, not QoS designs. The aggregate model is the default: one class scheme and identical markings across every VRF/tenant - voice is voice whichever virtual network it rides - which keeps the QoS design identical with or without virtualization and is the simplest to operate. The prioritized-VRF model instead ranks whole tenants (production VRF above guest VRF) and is justified only when the business ranks tenants, not applications. Two mechanics make either work end to end: encapsulation must carry the marking outward - routers reflect the inner ToS/DSCP into the outer GRE header and map it into MPLS EXP (the IP-tunnel cousin of the uniform/pipe tunneling modes above), so transit nodes can still classify multi-encapsulated traffic - and per-tunnel / per-VLAN H-QoS enforces the aggregate contract on the virtualized interface.

VXLAN obeys the same pattern with one Layer-2 trap. At encapsulation the VTEP copies the inner DSCP into the outer header (for bridged frames it derives the outer DSCP from the 802.1p CoS via a mapping) so the underlay can classify on the outer marking; at decapsulation the uniform vs pipe decision reappears - copy the possibly re-marked outer value inward, or preserve the original inner one. The trap: the VLAN header is not carried inside the tunnel, so a marking scheme living in CoS alone dies at the first VTEP - DSCP is the only marking that survives a VXLAN fabric end to end, which matters wherever L2 workloads are stretched over Overlays and BGP EVPN.

QoS as a security tool (scavenger containment)

The same machinery contains anomalies. Access-edge policers meter each endpoint against its normal baseline (the rate that covers ~95% of its legitimate traffic - a phone port has no business sending megabits); traffic beyond the baseline is re-marked to CS1 (scavenger), never dropped, because a burst may be legitimate. Every node then carries a deferential scavenger queue, so anomalous senders - worm outbreaks, DoS floods, runaway hosts - are squeezed out only when congestion appears, exactly when it matters, while normal operation is untouched. It is self-punishing containment: no signature, no false-positive outage, just economics at the queue (Infrastructure hardening - control, management, and data plane, Threat detection and mitigation - DDoS, spoofing, MITM).

Comparison

Recommendation / justification

Use DiffServ with a documented class model, mark at the access edge, and apply consistent end-to-end treatment (including the WAN/SP DSCP mapping - SPs often remark). Reconcile app budgets (latency/jitter/loss) against link speeds and buffers; remember microbursts are a buffer/QoS problem (see Factors affecting convergence).

What would change this

• Strict per-flow guarantees on a small scale -> RSVP/IntServ at the edge.
• SP remarks DSCP -> agree an SP class map and re-mark at the boundary.

IPv6 / dual-stack note

DSCP lives in the IPv6 Traffic Class field - mark and honor QoS identically for v6.

Related

• WAN transport architecture selection
• Factors affecting convergence
• Tunneling technology selection - QoS on encrypted overlays (per-tunnel QoS, pre-classify)

Spaced repetition

Voice per-class targets are ==<=150 ms one-way, <=30 ms jitter, <=1% loss== (interactive video ~200 ms, broadcast ~400 ms).

Sources

• Cisco Press, End-to-End QoS Network Design, 2nd Ed. (Szigeti et al.) - full-text pass: LLQ sizing, AF markdown, CAC, SP mapping, scavenger policing.
• Cisco Live BRKDCN-3953, End-to-End QoS Implementation and Operation with Nexus (overlay QoS, trust).

domain: Core · blueprint-ref: Core 5.6 QoS techniques and strategies · type: design-decision · status: complete · tags: [core, core/virtualization, tradeoff/latency, tradeoff/complexity]