Overlays and BGP EVPN

In one line

Decouple the logical network from the physical underlay via encapsulation (VXLAN / MPLS / GRE) plus a control plane (EVPN / LISP / flood-and-learn).

Encapsulation + control-plane options

BGP EVPN essentials

EVPN is the control plane that replaces flood-and-learn. Key route types: Type-2 (MAC/IP advertisement), Type-3 (inclusive multicast / BUM), Type-5 (IP prefix). It enables distributed anycast gateway, ARP suppression, integrated routing and bridging (IRB), and all-active multihoming (ESI). The underlay is usually eBGP (or an IGP) on a CLOS spine-leaf.

Inside the EVPN fabric - IRB semantics, gateways, and edge scale

The name-drops above have design-selecting mechanics. IRB comes in two flavours: with asymmetric IRB the ingress VTEP routes and bridges while the egress only bridges, so traffic between two subnets rides the destination's L2 VNI - which forces every VTEP to carry every VNI plus the remote ARP/MAC state, a scale penalty that grows with the fabric. Symmetric IRB has both ends route across a dedicated per-tenant L3 VNI, with the egress VTEP's router-MAC as the inner destination, so each VTEP holds only its local VNIs - the reason NX-OS implements symmetric and the reason large multi-tenant fabrics assume it. The distributed anycast gateway then puts the same gateway IP and one global anycast MAC on every leaf, so the first hop is always local and a mobility event never forces the endpoint to re-ARP - which is also what lets the fabric drop FHRPs entirely.

At the edge, vPC pairs present one anycast VTEP: a shared virtual IP (VIP) as the tunnel next hop plus a unique primary IP (PIP) per switch, on an NVE loopback deliberately separate from the routing-protocol loopback (a vPC consistency fault shuts the NVE source - shared loopbacks would take the underlay down with it). The trap: Type-5 prefix state is not synchronised between vPC peers, so external routes advertised behind the shared VIP can blackhole on an edge-link failure - the fix is advertising Type-5 routes from the PIP with the per-switch router-MAC (advertise-pip / virtual-rmac). Two more fabric-wide rules earn their keep: VTEPs forward tunnel traffic only from the BGP-learned VTEP allowed-list (with MD5 on the sessions), which quietly drops rogue-VTEP injection; and the control plane wants exactly two route reflectors - every extra RR is another copy of every route on every leaf. Cisco's recommended split remains IGP underlay + iBGP EVPN overlay - one protocol per function, decoupling underlay faults from the overlay; an all-eBGP fabric is supported, but RFC 7938's massive-scale-DC argument predates overlays and was never written for one. Finally, treat the border leaf as a dedicated role: it carries the external TCAM load, sits in the north-south chokepoint, and concentrates change risk - summarise or default toward the fabric, filter the host /32s and the fabric's own default outward, and hand off per-tenant VRF-lite.

SD-Access: the campus fabric (LISP + VXLAN + SGT)

SD-Access applies the overlay model to the campus using three separated planes. The control plane is LISP (RFC 6830) on control-plane nodes acting as the map-server / map-resolver, holding the endpoint-to-location (EID -> RLOC) database - so reachability is resolved on demand, not flooded. The data plane is VXLAN (UDP 4789): the VNI carries the virtual network and the SGT rides in the group-policy field of the header, so identity travels with the packet. The policy plane is SGT / group-based policy (SGACL), enforced independent of topology. Fabric roles: fabric edge nodes are routed-access switches that register endpoints to the control plane; border nodes join the fabric outward (an internal border to known domains like the DC, an external border to unknown/default like the internet, or anywhere for both - internal borders scale to as many as needed, while external and anywhere borders cap at four per site); control-plane nodes run the LISP mapping service. Segmentation is two-tier: virtual networks (VRFs) give macro-segmentation between groups, and SGTs (Infrastructure segmentation) give micro-segmentation within a VN. Fabric wireless puts APs into the fabric so wireless traffic is VXLAN-encapsulated at the fabric edge (the data path stays in the fabric), versus over-the-top APs that tunnel back to a WLC the old way. Design call: choose the fabric when you need policy-based segmentation, endpoint mobility, and an address plan decoupled from location at scale; stay traditional (Network hierarchy and topologies) when the extra moving parts (controller, map system, underlay) are not justified.

LISP beyond the fabric - mapping as a WAN tool

The fabric is one application; the mapping system itself is the design tool. Because an endpoint's EID never changes while its RLOC can, LISP buys three properties BGP-based designs pay dearly for. Destination-controlled ingress TE: the map-reply carries a locator-set with per-RLOC priority and weight, so a multihomed destination site dictates which of its attachments inbound traffic uses and in what ratio - inbound traffic engineering without AS-path prepends, communities, or more-specific leaks (BGP in enterprise design). Mobility: a workload keeps its EID across a move between sites or data centres; only the mapping updates, so addressing and sessions survive relocation. Address-family bridging: EID and RLOC families are independent, so v6 EIDs ride a v4 core (and every other combination) - a transition mechanism in its own right (IPv6 design and migration). Interworking with the non-LISP world runs through proxy xTRs (PxTR). The cost is the classic one: a pull-based mapping control plane in the path - first-packet resolution latency and a mapping-system dependency - which is exactly the complexity a plain BGP design avoids.

SD-Access control-plane evolution - Pub/Sub and the fine print

Two LISP control-plane generations now coexist. The original LISP/BGP model redistributes between LISP and BGP at the borders; the newer LISP Pub/Sub has borders subscribe and the control-plane node push updates - no LISP-to-BGP redistribution, faster convergence, and a simpler protocol stack - and is the recommendation for new fabrics. Pub/Sub also unlocks border behaviours the old model could not express cleanly: a dynamic default border and LISP backup internet, so egress re-steers itself when a border or its external routing changes (an external border in Pub/Sub still needs a real default in its RIB to anchor on). Steering stays two-knobbed: border priority shapes egress preference, while AS-path prepend shapes ingress - distinct levers, distinct failure questions. On the control-plane node itself, the sizing fine print matters more than the role: a fabric WLC synchronises client state with at most two control-plane nodes, so a site running fabric wireless caps at a pair; colocated border-plus-CP is the sensible default, but a high-roam environment justifies a dedicated CP pair because every roam is a control-plane update burning CPU. And the underlay rules are stricter than they look: Layer 3 routed access with MTU 9100 (1550 minimum for the 50-byte VXLAN tax), BFD, IS-IS if LAN-automated - and RLOC /32 reachability must not rely on a default route; APs equally need a specific route to the WLC. Dual stack lives in the overlay only - the SD-Access underlay stays IPv4.

The fabric's management plane - controllers, budgets, and the LCD

The controller and identity tier is a design object with its own resiliency ladder and its own latency contract. DNA Center deploys as a standalone node, as a three-node HA cluster (members on a 10 Gbps interconnect under 10 ms RTT - node failure detected in about five minutes, service restored in 7-13 with automatic failback), or in disaster recovery: clusters at a main and a recovery site plus a witness VM at a third site, joined over 1 Gbps under 350 ms with IPsec - a site failure is detected in about three minutes, failover takes 15-30 minutes with service down for up to 30, and failback is a manual decision. The split is the classic one: HA answers node failure inside a site, DR answers losing the site - and the wider latency envelope plus the human-in-the-loop failback are what the geography costs.

Placement then follows the round-trip budgets, because each one sits inside a control-plane operation: about 200 ms RTT is the ceiling between DNA Center and its fabric devices (execution already degrades past ~100), about 300 ms between DNA Center and ISE (degrading past ~200), ISE personas in a distributed deployment tolerate about 300 ms between one another, and ISE to its network devices is about 100 ms because the exchange is RADIUS - authentication, authorisation, and TrustSec policy download all ride it (AAA and identity - RADIUS, TACACS+, 802.1X). The wireless tail is tighter still: ~100 ms control-plane node to WLC and ~20 ms WLC to AP (Controller placement, HA, and tunnel optimization). The budgets decide the shared-services pattern: a local DC or services block keeps everything trivially inside them, while a remote-DC design centralises DNA Center and the ISE administration persona and pushes distributed policy nodes out toward the metro sites, so the 100 ms RADIUS budget is met by locality rather than luck (and the controller itself expects Internet reachability - an air-gapped estate must design around that explicitly).

Sizing, finally, is a least-common-denominator exercise across every solution element, because each caps a different dimension: the controller bounds devices, endpoints, fabric sites, and IP pools and VNs per site; ISE bounds concurrent active endpoints, SGACLs, and SXP peerings; a control-plane node bounds VNs and host entries; a border bounds external prefixes and security-group enforcement state (SXP per VN, IP-SGT bindings, SG-ACEs); an edge bounds host entries and fabric access-tunnels. The fabric's effective scale is the smallest of these, checked per platform - the same LCD discipline the WAN applies to transport features (WAN transport architecture selection).

Fabric role placement by site size

The three roles (control-plane, border, edge) scale by collocation vs separation. In a small or remote site you collocate them - control-plane and border on the same box, or a single fabric-in-a-box doing all three - trading redundancy for cost and simplicity where the endpoint count does not justify dedicated nodes. As a site grows you separate the roles onto dedicated devices and make the control-plane and border nodes redundant, since they are now shared points of failure for endpoint resolution and egress. Very small or far-flung sites can instead run over-the-top or reach a larger fabric across a transit - SD-Access transit preserves SGT/VN end-to-end, while plain IP transit drops that context and forces policy to be re-derived at the far side. The recurring design call is the familiar simplicity/cost vs scale/availability trade applied to where the fabric roles live, not which protocols run. Between one big site and many sites plus a transit sits an intermediate rung. A single fabric site stretched across buildings has three well-known pains - one subnet everywhere (any pool spans all buildings), one failure domain, and the site's IP-pool, border, and control-plane scale ceilings. Fabric zones relieve the first and third without splitting the site: a zone is a child site of the parent fabric site that scopes L3VN-and-IP-pool provisioning to a subset of edge nodes - pools are created at the parent, then assigned to one or more zones - so a building's addressing (and the blast radius of its pools) stays local while the site keeps one control plane. The membership rules are strict: only edge-class nodes join a zone (edge, extended, and policy-extended nodes - a box collocating border or embedded-WLC roles cannot), and an extended node must sit in the same zone as its parent edge. When the failure domain or the control-plane ceiling itself is the pain, zones stop helping - that is the cue for separate fabric sites and a transit (below).

Virtual networks are isolated by construction, so the design must answer where they meet: shared services (DHCP, DNS, NTP, ISE, the controller, WLCs) live outside the fabric, upstream of the border, and a fusion device - a VRF-aware router or firewall peered per-VN from the borders - leaks routes between the fabric VNs and wherever those services sit (the global table, for selective and granular leaking, or a dedicated shared-services VRF via route-targets). Making the fusion a firewall turns the leak point into the inter-VN policy and inspection choke point (Perimeter security - firewalls and IPS-IDS); the modern alternative, LISP extranet, provides the same shared-services reachability inside the fabric without BGP leaking. Two sizing corollaries follow: keep the VN count small - every VN costs a VRF-lite handoff and a BGP session on every border - and remember the ~50 bytes of VXLAN overhead in the underlay MTU plan (MTU, fragmentation, and PMTUD). Legacy segments that must bridge into the fabric use a Layer 2 border/handoff - supported, but it reimports the L2 risks the fabric removed, so treat it as a migration tool, not a design pattern.

Multisite SD-Access - transits, anchored VNs, and multiple controllers

The moment a second fabric site exists, the design question is what survives the trip between them - the VN, the SGT, both, or neither. Four transit families answer differently.

Plain IP transit is a per-VN VRF-Lite handoff at every site's border into any routed WAN: macro-segmentation survives as VRFs, but the SGT does not - it must be rebuilt from IP-to-SGT bindings distributed by SXP through ISE (Infrastructure segmentation) - and every site keeps a fusion device that no controller automates. The same transit can carry both tiers end to end if the borders and the path run CMD inline tagging: macro and micro segmentation then cross the WAN intact, priced in TrustSec-capable hardware along the whole transit path and jumbo MTU on a P2P or Metro-E service - the tag is a Layer 2 artefact, so the WAN must behave like a wire.

SD-Access transit is the native answer: the VXLAN header itself carries the VN (VNID) and the SGT between sites, so segmentation needs no re-derivation, the controller automates the interconnect, and the per-site fusion disappears. Its machinery is a set of transit control-plane nodes - dedicated devices that sit outside the forwarding path, need only plain IP reachability to every site's borders, and hold aggregate prefixes of every fabric site so inter-site resolution stays a lookup, not a flood. Two prerequisites gate it: a site joins through an external or anywhere border (never internal-only), and the WAN must carry the VXLAN MTU - the tunnel cannot fragment - with TCP adjust-MSS as the confessed fallback that rescues only TCP. The transit runs under either control-plane generation, but the fine print favours the new one: Pub/Sub supports four transit control-plane nodes to LISP/BGP's two, and multicast across the transit arrives only for Pub/Sub fabrics.

SD-WAN transport splits into two postures. Independent domains keep the campus and WAN fabrics separate and stitch them the old way - a VRF-Lite plus inline-tagging handoff at each WAN edge - so two controllers each manage their own overlay and the seam between them is yours to keep consistent. The integrated domain collocates border and control-plane on the WAN edge itself and has the two management planes exchange the VN-to-VPN mapping directly, removing the seam - and it is the one design that still mandates the LISP/BGP control plane, so choosing it is choosing against Pub/Sub's border innovations.

The anchored VN completes the toolset. A dedicated guest VN is the base pattern - its own VRF for control- and data-plane isolation, handed off at the border by BGP straight into a DMZ firewall. Multisite Remote Border (MSRB) generalises the classic WLC guest anchor (Roaming - L2 vs L3 mobility) to any VN: the VN is anchored to a different fabric site's border and control-plane pair - typically the DMZ site - and the anchoring site's edge nodes VXLAN-encapsulate straight to that remote border, so the VN's egress and its subnet exist only where policy wants them, whichever site the user joins from. Two costs are mechanical: the no-fragmentation MTU rule now spans sites, and a fabric WLC must synchronise with the anchor site's control-plane nodes as well as its own - a Catalyst 9800 syncs with up to eight CP pairs where AireOS stopped at two, which bounds how many anchored sites one WLC generation can serve. (Fabric wireless itself keeps CAPWAP for AP control while client data is VXLAN-switched at the edge; over-the-top wireless uses the fabric as plain transport - the redundancy fork's fabric behaviour lives in Controller placement, HA, and tunnel optimization.)

When one controller cluster is no longer enough - scale, geography, or organisational boundaries - the multi-cluster model attaches up to five DNA Center clusters to a single ISE deployment in an author/reader split: exactly one cluster authors the group-based policy objects (virtual networks, groups, contracts) into ISE and every other cluster reads them - the mechanism behind globally consistent group-based policy across an estate too big for one controller. The fabric sites of different clusters then interconnect like any others: a shared SD-Access transit spanning clusters keeps VN and SGT native (with LISP extranet or VN policy answering shared services at that scale), or SD-WAN and IP transits where the domains must stay independent.

Multicast in the campus fabric

The fabric automates overlay multicast for ASM and SSM, with sources, receivers, and the ASM RP each free to sit inside or outside the fabric and per-VN RPs supported - RP placement stays the classic exercise (Multicast routing design - PIM, RP, MSDP). The deployment fork is the overlay-BUM lever wearing fabric numbers: head-end replication needs no underlay multicast and suits roughly fewer than ten edge nodes in a site, paying with replication load on the head-end; native multicast builds an underlay PIM tree, offloads the head-end, and is the at-scale choice - but it carries no IPv6 overlay multicast, so a dual-stack overlay that needs v6 multicast is forced onto head-end replication regardless of edge count. And multicast originally did not cross the SD-Access transit at all - the capability lands as a Pub/Sub-only enhancement (alongside multiple RPs per VN, group-to-RP mappings, and concurrent ASM/SSM), one more weight on the Pub/Sub side of the control-plane scale.

Scaling and interconnecting fabrics: super-spine and EVPN multisite

Two questions arise once a single spine-leaf fabric is no longer enough. Scaling out: a 3-stage Clos (leaf-spine) is bounded by spine port count, so to grow you add a super-spine tier (a 5-stage Clos / pods) - leaves connect to spines within a pod, spines connect to super-spines between pods - preserving non-blocking any-to-any while scaling horizontally. Interconnecting fabrics (DCI): to join multiple VXLAN EVPN fabrics (separate DCs or pods) you either stretch one fabric across sites or use EVPN multisite. Stretching makes the sites one failure domain - a BUM storm, a control-plane fault, or a misbehaving host propagates everywhere. EVPN multisite instead places border gateways (BGW) at each site edge: each fabric keeps its own control plane and BUM handling, and the BGWs re-originate EVPN routes and stitch the overlays at the boundary, so the failure domain stops at each site while end-to-end L2/L3 reachability is preserved. The BGW is also the clean control point for cross-site BUM (designated-forwarder election, storm control) and for per-site policy. The call mirrors the vault's modularity theme - isolate failure domains at the interconnect rather than melting fabrics into one; the DCI itself should ride diverse, resilient transport (Transport resiliency, diversity, and migration), and the BUM replication choice (multicast vs ingress) follows Multicast routing design - PIM, RP, MSDP.

EVPN and multicast - from BUM flooding to mVPN drop-off

EVPN's default treats customer multicast as just the M in BUM: ingress replication floods it to every PE in the instance regardless of receivers - acceptable for ARP-scale noise, wrong for real streams. The first fix makes the fabric IGMP-aware: IGMP snooping at the edge plus EVPN routes that synchronize join/leave state between the multihomed PEs of an Ethernet segment (join-sync and leave-sync route types), so the DF forwards while the non-DF holds hot state and an all-active pair neither duplicates nor blackholes on failover. The second fix, at scale, is the mVPN drop-off: stitch the EVPN edge to a Profile 14 mVPN core so multicast rides on-demand mLDP trees only to the egress PEs that actually have receivers instead of being flooded fabric-wide; both PEs of a multihomed pair pull the tree - a live-live protection scheme - and only the DF forwards onto the segment, with IRB placing the multicast routing inside or outside a VRF as tenancy demands (Multicast routing design - PIM, RP, MSDP).

Evaluating a DCI - the option families and the checklist

Before arguing technologies, sort the candidates into their three families, because each family changes who operates what: physical/logical connectivity (dark fibre, DWDM waves, L2 trunks, routed point-to-point links - maximum control and the substrate everything else rides, at facilities cost and distance limits, Optical transport - CWDM vs DWDM); SP-provided solutions (MPLS L3VPN, MPLS L2VPN/pseudowire, VPLS - the provider operates the interconnect, you inherit its SLA and its limits, MPLS L3VPN and L2VPN); and customer-run overlays (EVPN-VXLAN, GRE/IPsec and friends - you operate the interconnect over whatever transport exists). Beware the fourth bucket: tools that look adjacent but are not DCI options at all - a plain tunnel or an intra-DC fabric protocol does not become an interconnect design just because it can carry a VLAN somewhere.

Then score the survivors against a grouped checklist rather than a feature shoot-out - business (continuity, efficiency/cost of the outcome), architectural (L2 extension, path optimisation, ease at three-plus DCs - several classic options carry a two-site limit - multi-tenancy, QoS transparency, geo-distributed clusters, transport independence), operational (maintenance burden, rebalancing workloads between sites), and technical (multihoming with loop avoidance, multipathing, scalability, complexity, cost, workload mobility, DR, site migrations). And read the qualifiers as hard as the ticks: a capability that "depends on the underlay" or "needs underlay support" is a dependency to design, not a feature you own - the difference between a glossy comparison cell and a deployable answer. This framing slots straight under the multisite doctrine above: the checklist picks the family and option; the failure-domain rule decides how it terminates.

One late entrant joins the customer-run family: end-to-end SRv6, which removes the stitching function rather than optimising it - DC overlay and WAN share one IPv6 data plane, so the border-gateway/hand-off tier the other options must design does not exist (Segment Routing); its checklist costs are ecosystem maturity and host/edge data-plane readiness.

Why it matters for design

VXLAN + EVPN is the dominant DC and SD-Access fabric, and the same CLOS + VXLAN concepts underpin AI fabrics (Fabric transport - Ethernet vs InfiniBand vs UEC). Know control-plane learning vs flood-and-learn and the three core EVPN route types.

Why EVPN over VPLS

EVPN is the modern replacement for VPLS (MPLS L3VPN and L2VPN) because it moves MAC learning into a BGP control plane instead of data-plane flood-and-learn. The design wins: all-active multihoming (versus VPLS active/standby), ARP suppression and reduced BUM flooding, faster convergence, and better scale - which is why EVPN (over VXLAN or MPLS) is the default for new DC and DCI fabrics.

IPv6 / dual-stack note

EVPN advertises both IPv4 and IPv6 host routes (Type-2/5); VXLAN encapsulates either. Underlay can be v6. SD-Access native-mode overlay multicast is v4-only - v6 overlay multicast forces head-end replication.

Related

• MPLS L3VPN and L2VPN
• Infrastructure segmentation
• BGP in enterprise design

Spaced repetition

Sources

• Cisco Live BRKENT-2076 (M. Nagireddy, APJC 2022, user-supplied PDF, all 59 pages passed 2026-07-26): multisite transit family + matrix, transit control-plane nodes, border caps, fabric zones, MSRB, fabric multicast modes, DNA Center HA/DR and latency budgets, multi-cluster author/reader.
• Cisco Live BRKENT-2004 (CLUS 2022): EVPN multicast-as-BUM, IGMP join/leave sync between multihomed PEs, mVPN drop-off with live-live pull.
• Cisco Live BRKSP-2551 (2026), Introduction to Segment Routing - A foundation for Autonomous Networks (Donzelli, Cisco / Gorovoy, Nebius; incl. the Microsoft AI-fabric and Nebius DCI material)
• Cisco, VXLAN Network with MP-BGP EVPN Control Plane design guide + Nexus 9000 VXLAN BGP EVPN D&I white paper (bonus batch, fetched) - IRB semantics, DAG, vPC VTEP, RR/underlay doctrine, border-leaf scale.
• Cisco, SD-Access Solution Design Guide (CVD, bonus batch, fetched; truncated at extended nodes) - LISP Pub/Sub, border steering, CP-node and underlay fine print.
• Zigbits (Z. Zsiga) / Agile IS (M. Radwan), DCI Design Comparison Chart (2020, bonus content) - option-family taxonomy and the business/architectural/operational/technical evaluation checklist.
• Cisco Live BRKRST-2041 (2013): LISP EID/RLOC mechanics, destination-controlled ingress TE, mobility and transition use cases.
• Cisco Press, Building Data Centers with VXLAN BGP EVPN; EVPN RFCs.
• VXLAN EVPN multisite / border-gateway and multi-tier Clos (super-spine) DC design (Cisco VXLAN BGP EVPN; RFC 8365).

domain: Core · blueprint-ref: Core 5.2.g/5.2.h Overlay encapsulation and BGP EVPN · type: concept · status: complete · tags: [core, core/virtualization, tradeoff/scale]