BGP in enterprise design
Requirement / business driver
Use BGP where the requirement is policy control, scale beyond an IGP, internet multihoming, or a DC/WAN fabric - not as a general interior protocol.
Where BGP earns its place
- Internet edge / multihoming - provider-independent policy and failover.
- DC underlay - eBGP per-rack ASN (simple, scalable) and EVPN as the overlay control plane (see Network Virtualization batch).
- WAN / SD-WAN - reachability + policy across many sites.
- Scale - carries full tables and huge prefix counts an IGP should never hold.
iBGP scaling: full mesh vs RR vs confederation
iBGP requires a logical full mesh (n(n-1)/2 sessions) because iBGP routes are not re-advertised to other iBGP peers. Two ways to break the mesh:
| Option | How it scales | Best fit | Watch-outs |
|---|---|---|---|
| Route Reflectors (RR) | RR re-advertises iBGP routes to clients | Enterprise default - simplest | Path hiding (use add-paths); place RRs redundantly; cluster-id design |
| Confederations | Split AS into sub-ASes | Very large / SP networks | More complex; rarely needed in enterprise |
eBGP traffic engineering (multihoming)
- Outbound control with local-preference.
- Inbound control with AS-path prepend, communities to upstreams, and advertising more-specifics.
- MED between a single neighboring AS; weight is Cisco-local.
Multihoming and asymmetric routing
A dual-edge / dual-ISP design can send a flow out one edge and receive the return via the other - which breaks stateful firewalls: the second site's firewall never saw the outbound packet, has no session state, and drops the return (asymmetric paths break stateful flows entirely, worst of all during failover). Fixes, used together:
- iBGP between the edge routers (direct link or GRE) plus policy that makes each site's own prefixes preferred via its own link, keeping flows symmetric.
- Organised advertisement - each site advertises its more-specifics (longest match; e.g. a /16 split into two /17s) alongside the aggregate, so return traffic prefers the originating site (same idea for IPv6).
- NAT case - if the edge firewalls NAT, peer the distribution/core directly with the edge routers so NAT stays consistent.
Treat symmetry as a design requirement wherever stateful devices (Perimeter security - firewalls and IPS-IDS) sit in the path.
Securing BGP
Peer authentication (TCP-AO, legacy MD5), GTSM/TTL-security, max-prefix limits, prefix/bogon filtering, uRPF, and RPKI/ROA origin validation at the internet edge.
BGP as the enterprise core
Most enterprises run an IGP (OSPF/IS-IS) in the core, but a very large or global enterprise can hit IGP scalability limits (huge prefix counts) or need distributed administrative control and rich policy across regions. The pattern: split the network into multiple IGP islands (failure domains) glued together by BGP, hiding topology/reachability at the island edges. Benefits: better scale (fewer prefixes per region), fault isolation (instability in one domain does not leak to others), per-region administrative control, and flexible intra/inter-domain policy (e.g., MP-BGP + MPLS for path separation, smooth IPv6-over-IPv4 core). Caveats: BGP convergence is slower than an IGP, and BGP path selection is policy-based, not shortest-path - to follow the physical shortest path over the core you must lean on the IGP metric / AIGP or careful policy.
Don't become a transit AS
A multihomed enterprise must advertise only its own prefixes to its providers - never re-advertise routes learned from one ISP to another, or you become a transit AS and carry the two providers' traffic through your network (saturating your links and exposing you). Safeguards: an outbound prefix-list / AS-path filter permitting only your own aggregates, and tagging received routes no-export so they stay local. This is the inverse of the traffic-engineering above - there you steer your own traffic, here you prevent other people's. Check it explicitly ("are we advertising anything we didn't originate?"), because accidental transit is silent until your edge links saturate.
Route reflector redundancy and topology alignment
A single RR is a single point of failure, so production designs cluster two or more RRs for the same clients - but the cluster-id choice is subtle. RRs in the same cluster carry the CLUSTER_ID attribute and ignore a route received from a peer RR with the same cluster-id (loop avoidance). The side effect: if two redundant RRs share a cluster-id, a client reachable through both can lose a prefix when one RR fails, because the survivor discarded the other's copy. Giving each redundant RR a different cluster-id keeps both copies alive (the more resilient choice); ORIGINATOR_ID and CLUSTER_LIST then prevent loops across the distinct clusters. The larger trap is topology incongruence. An RR only reflects the single best path it itself chose, so when the iBGP-RR (logical) topology does not match the physical/IGP topology, a client can be handed a next-hop that is not reachable the way the RR assumed - producing a forwarding loop (e.g. two data-center routers each forward toward the other's exit and loop between them). Keep logical and physical congruent: align each client to the RR that is physically/IGP-closest (simplest), or add the missing links / use separate clusters with a direct RR-to-RR session. RR placement is therefore a topology decision, not only a scaling one, and incongruent designs can still yield suboptimal routing that only add-paths or careful placement resolves.
Two further RR design levers close out the toolkit. Where the RR computes from - reflection has a vantage-point problem beyond congruence: the RR runs best-path from its own IGP position, so every client inherits the exit that is closest to the RR - and the moment RRs are centralised (a control-plane pair in one DC, or a virtual RR farm, which is exactly where dedicated RRs gravitate because the role needs CPU and memory, not ports) hot-potato routing collapses toward the RR's location. Optimal Route Reflection (ORR) fixes the computation instead of the placement: using the link-state IGP's full topology view, the RR computes the next-hop metric from each client's (or client group's) vantage and reflects the path that client would itself have chosen - restoring per-client hot-potato from a centralised RR, at the price of requiring OSPF/IS-IS visibility and extra RR CPU. It is the third answer to reflection distortion, alongside congruent placement and add-paths.
What else the RR does - a dedicated RR is control-plane only: kept out of the forwarding path, its BGP routes suppressed from RIB/FIB install, sized on session and prefix count, and free to live wherever placement or ORR logic allows. An inline RR doubles as a forwarding node (ABR/ASBR), and that duality is load-bearing in unified/seamless-MPLS designs: area-border inline RRs re-advertise BGP-LU PE loopbacks with next-hop-self per area segment, stitching an uninterrupted end-to-end LSP across areas that share no IGP view (MPLS fundamentals). Choose dedicated for scale and blast-radius isolation; inline where the topology needs the RR to also anchor the label plane.
Dual-carrier fine print - transit election, carrier preference, failback
Three traps live in the dual-carrier MPLS design that a single carrier never meets. First, someone must be transit - deliberately. With every site filtered to advertise only its own prefixes, a single-homed site on carrier A has no path to a dual-homed site whose A attachment just failed: the route must cross carriers somewhere. The pattern is elected transit sites - a few sites (typically large hubs) that deliberately re-advertise between the two provider clouds, chosen for spare capacity (transit traffic rides their links) and geographic diversity (so cross-carrier detours stay short), while every other site keeps the non-transit filter. Transit becomes a designed role with a capacity budget, not an accident.
Second, keep carrier preference inside BGP. When the CE pair exchanges what each carrier taught it over iBGP, primary and secondary path selection stay attribute-driven (local-preference per carrier, and so on). Re-advertising both WAN paths through the site IGP instead flattens the BGP attributes away - remote routers see two equal-cost IGP routes and split traffic across carriers you meant to rank.
Third, failover is easy - failback is the trap. In the classic MPLS-primary / IGP-backdoor-backup design (eBGP AD 20 beats the overlay IGP's external AD, so preference and failover both work by default), trouble starts after the primary is restored: during the outage the site's prefixes were redistributed into BGP locally, and locally originated routes carry weight 32768 - which beats the returning eBGP path's default weight of 0, so the router never reverts. Deterministic reversion needs the preference made explicit - pin the eBGP session's weight above 32768 (or use local-preference) so restoration wins by policy, not by hoping defaults align. The general lesson for any mutually-redistributed dual-path design: state the reversion policy explicitly; default tie-breakers were never designed for it.
iBGP next-hop policy and session fall-over
eBGP hands iBGP a hidden dependency: the NEXT_HOP of an eBGP-learned route is passed unchanged to iBGP peers, so every interior router must be able to resolve an external next hop it has no adjacency with. Two designs exist. next-hop-self at the border rewrites the NH to the border router's loopback - the edge subnets stay out of the IGP, interior convergence follows one well-known loopback, and the border becomes the abstraction point. Carrying the edge links in the IGP instead preserves the true exit in every RIB (finer-grained TE, faster indirection-free failover to an alternate exit) at the price of leaking external topology into the interior. Most enterprise designs take next-hop-self and keep the IGP clean.
Session liveness is its own design surface. Fast external fall-over (default) tears a directly connected eBGP session the instant the shared interface drops - no hold-time wait. For everything else - iBGP, loopback-peered eBGP - per-neighbor fall-over ties the session to the RIB: the moment the route to the peer's address disappears, the session drops, with no hold-down. That converts BGP convergence into an IGP property, and it cuts both ways: the IGP must supply an immediate alternate (think LFA, Fast convergence techniques) or a transient underlay flap becomes a full BGP session reset and table churn. Aggressive fall-over belongs where alternates exist; where they don't, the slower keepalive/hold-time detection is the stability choice - the same route-around versus forward-through judgement as everywhere else in convergence design.
Update economics and conditional advertisement
Two more scale/TE levers round out the toolkit. BGP update generation scales with distinct outbound policies, not with peer count: peers that share an outbound policy can be grouped so each update is built once per group and replicated, and iBGP - where everyone receives the same view - is the natural grouping; this attacks the work per update, complementing route reflectors and confederations, which attack the number of sessions. And for inbound failover there is conditional advertisement: advertise a prefix to one upstream only while a watched route is absent (exist/non-exist logic), so the backup path stays invisible until the primary actually dies - inbound failover without a permanently visible backup that the Internet might prefer at the wrong moment. It is the deliberate, event-driven cousin of the leak-more-specifics TE above.
The update-group economy has a failure mode: the slow peer. Replication means the group advances together, so one member that cannot drain its update queue - starved CPU, poor TCP throughput, a congested path - holds the formatted updates in memory and stalls generation for every healthy member of the group. The design answer is detect-and-quarantine: watch per-peer output queues, and move a persistently slow peer into its own update group so the rest converge at full speed while the straggler catches up alone. The companion pacing knob is MRAI (minimum route advertisement interval, applied per destination): eBGP defaults to ~30 s, iBGP to 0. Raising iBGP MRAI to a few seconds toward RR clients that do not need instant updates damps churn and packs more routes per update at a small convergence cost - the BGP instance of the aggressive-vs-stable timer trade (Fast convergence techniques). Session liveness follows the same philosophy: leave keepalive/hold timers alone and let BFD own detection, with PIC owning the repair.
Recommendation / justification
Introduce BGP only where policy, scale, or multihoming demand it; for iBGP scale use route reflectors in the enterprise (reserve confederations for SP-scale). The justification is always the requirement (policy/scale/multihoming), never "because we can."
IPv6 / dual-stack note
MP-BGP carries IPv4 and IPv6 as separate address families over one session - design both AFIs and apply the same security (auth, max-prefix, RPKI) to each.
Related
- IGP selection - OSPF vs IS-IS vs EIGRP
- Securing routing protocols
- Route aggregation and summarization
- Metric-based traffic engineering
Spaced repetition
Why does iBGP need a full mesh (or RR/confederation)?
An iBGP speaker does not re-advertise routes learned from one iBGP peer to another iBGP peer, so every speaker must hear them directly.
In the enterprise, the simpler way to scale iBGP is [...] (confederations are for very large / SP networks).
In the enterprise, the simpler way to scale iBGP is route reflectors (confederations are for very large / SP networks).
For inbound traffic engineering on a multihomed edge, the main BGP levers are [...].
For inbound traffic engineering on a multihomed edge, the main BGP levers are AS-path prepend, communities, and advertising more-specifics.
MP-BGP carries IPv4 and IPv6 by using [...].
MP-BGP carries IPv4 and IPv6 by using separate address families (AFIs) over one session.
When would an enterprise use BGP (not an IGP) as the core routing protocol?
When the IGP hits scale limits or regions need distributed admin control/policy - split into IGP islands glued by BGP (better scale, fault isolation, per-region control); caveats: slower convergence and policy-based (not shortest-path) selection, so use AIGP/IGP metric for the shortest path.
Why can Internet multihoming with two sites and per-site stateful firewalls break traffic, and how is it fixed?
Return traffic can arrive via the other site's link, whose firewall has no session state and drops it (asymmetric routing breaks stateful flows). Fix with iBGP between edges + policy, more-specific per-site advertisements, and consistent NAT peering.
How does a multihomed enterprise avoid becoming a transit AS?
Advertise only its own prefixes to providers (outbound prefix-list / AS-path filter permitting just your aggregates) and tag received routes no-export - never re-advertise one ISP's routes to another.
Why is accidental transit dangerous and hard to spot?
Your AS would carry both providers' through-traffic, saturating your edge links - and it stays silent until they congest, so it must be checked explicitly in the advertisement policy.
Two redundant RRs sharing the same cluster-id - what breaks on failure?
They use CLUSTER_ID to ignore each other's routes (loop avoidance), so a client reachable via both can lose a prefix when one RR fails; give redundant RRs different cluster-ids for resilience.
Why can a route-reflector design form a forwarding loop?
If the iBGP-RR (logical) topology is incongruent with the physical/IGP topology, the RR reflects a best path whose next-hop isn't reachable as assumed; fix by aligning each client to its physically/IGP-closest RR.
Dual carrier with non-transit filters everywhere: what breaks, and what is the fix?
A single-homed site on carrier A cannot reach a dual-homed site that lost its A link - no path crosses carriers. Elect a few transit sites (spare capacity, geographically diverse) that deliberately bridge the clouds, while all other sites stay non-transit.
After the MPLS primary is restored, the CE keeps using the IGP backdoor because [...] - fix by [...] for deterministic reversion.
After the MPLS primary is restored, the CE keeps using the IGP backdoor because the locally redistributed route carries weight 32768, beating eBGP's default 0 - fix by pinning the eBGP peer's weight above 32768 (or using local-preference) for deterministic reversion.
Why run iBGP between the CE pair instead of re-advertising WAN routes through the site IGP?
IGP re-advertisement flattens BGP attributes - remotes see equal-cost paths and split across carriers; iBGP keeps primary and secondary carrier preference attribute-driven.
What problem does next-hop-self solve at the BGP border?
eBGP NEXT_HOPs pass unchanged into iBGP, so interior routers would need routes to external subnets; rewriting the NH to the border loopback keeps edge links out of the IGP and makes interior convergence follow one well-known address.
Why can per-neighbor BGP fall-over make things worse on a fragile underlay?
It drops the session the moment the route to the peer leaves the RIB, with no hold-down - without an immediate IGP alternate (LFA), a transient flap becomes a full session reset; use it where alternates exist, keepalive/hold-time detection where they don't.
BGP update generation cost scales with [...] - group same-policy peers (iBGP naturally) so each update is [...], complementing RR/confederations which reduce session count.
BGP update generation cost scales with distinct outbound policies, not peer count - group same-policy peers (iBGP naturally) so each update is built once per group, complementing RR/confederations which reduce session count.
What does BGP conditional advertisement buy for dual-homed inbound failover?
The backup prefix is advertised to the second upstream only while a watched (primary) route is absent - the backup stays invisible until the primary dies, so no permanently visible backup path can be preferred prematurely.
What problem does Optimal Route Reflection (ORR) solve, and what does it require?
A centralised or virtual RR computes best-path from its own IGP vantage, so clients inherit the RR's closest exit and hot-potato collapses toward the RR; ORR computes the next-hop metric from each client's vantage using the link-state IGP's full view (OSPF/IS-IS required) - the third fix alongside congruent placement and add-paths.
One [...] stalls its whole BGP update group (replication advances together), so persistently slow peers are detected and quarantined into [...].
One slow peer stalls its whole BGP update group (replication advances together), so persistently slow peers are detected and quarantined into their own update group.
Sources
- Cisco Live BRKMPL-2103 (2026): ORR, dedicated vs inline RR (seamless-MPLS BGP-LU), slow-peer quarantine, MRAI defaults and pacing.
- White/Slice/Retana, Optimal Routing Design (full pass) - peer-group update economics, conditional advertisement.
- Cisco Press, CCIE Routing and Switching v5.0 OCG Vol 2, 5th Ed. - NEXT_HOP handling, fast external fall-over, per-neighbor fall-over.
- Cisco Live BRKRST-2041 (2013): elected transit sites, iBGP carrier preference, weight-32768 failback trap.
- Cisco Press, Optimal Routing Design; Routing TCP/IP Vol II.