Skip to content

Building Blocks

Physical Components

You need some switches/routers with connections between them: 

flowchart LR
    A(Switch 1)
    B(Switch 2)
    A ---- B
    B ---- A
flowchart LR
    A(Switch 1)
    B(Switch 2)
    C(Switch 3)
    A --- B
    B --- C
    C --- A
flowchart LR
    A(Switch 1)
    B(Switch 2)
    C(Switch 3)
    D(Switch 4)
    A --- B --- C --- D --- A
flowchart LR
    A(Switch 1)
    B(Switch 2)
    C(Switch 3)
    D(Switch 4)
    A --- B --- C --- D --- A
    B --- D
    A --- C

Topology is arbitrary (within reason).

Logical Components

Each link between switches is configured with an IP subnet, usually /30 or /31

flowchart LR
    A(Switch 1)
    B(Switch 2)
    A -- 192.168.0.0/31 --- B
    B -- 192.168.1.0/31 --- A

Loopback addressing

Each switch is configured with a loopback IP address (or sometimes two, depending on the platform)

flowchart LR
    A(Switch 1\n172.16.0.0/32)
    B(Switch 2\n172.16.0.1/32)
    A -- 192.168.0.0/31 --- B
    B -- 192.168.1.0/31 --- A

Underlay Routing Protocol — Interior Gateway Protocol

The underlay IGP's function is to enable all of the switches to reach every other switch's loopback address.

IGP options:

  • OSPF: easy button, works for small networks, limited control
  • IS-IS: also easy, works well for small or large networks, less common in enterprise, better control
  • BGP: not quite as easy, most flexible
  • Static routes: possible for two-node networks with a single logical connection (n links in LAG)

Overlay Routing Protocol — MP-BGP

Each switch participating in the overlay needs to exchange BGP routes with the other participants. BGP sessions are established between the routers' loopback interfaces, which are reachable thanks to
the underlay routing protocol.

It's typical to use iBGP in the overlay (all routers configured with the same autonomous system number). However, with iBGP we require either full mesh of BGP sessions or route reflection.

In small networks, forming a full BGP mesh is a good approach.

In larger networks, it may be helpful or even necessary to use a pair of route reflectors instead of a full mesh. The route reflectors can be routers in the forwarding path or out of band, perhaps running on VMs.

Note

This might look like a spine-leaf network, but it is not (necessarily). The links shown represent where overlay BGP sessions are formed and do not represent the physical (underlay) connections between routers.

Route reflectors become helpful in larger networks for several reasons:

  • The number of BGP sessions required for a full mesh is equal to n2 − n, whereas with route reflectors the number is about 2n. It's much easier to configure, manage, and monitor BGP sessions on an order of n than n2.

  • Route reflectors also reduce the memory requirements of each router by reducing the number of paths each router must store. Because the best path selection algorithm runs on the route reflector, the clients only receive the best path, rather than each router storing all of the paths and running its own best path selection. The inherent downside is that the best path from the RR's point of view may not always be the best path for every router.