Underlay routing protocols for data centers

When it comes to choosing the routing protocols that ought to drive the data center, we should differentiate between the overlay and underlay protocols. On the one hand, the overlay network is a virtual network which is routed on top of underlay network infrastructure, with the routing decision taking place with the help of software, whereas the underlay Network is the physical infrastructure above which overlay network is built, responsible for the delivery of packets across networks.

One of the most common encapsulation protocols for overlay networks in data centers is VXLAN or Virtual Extensible LAN, widely deployed in many layer 3 data centers to provide layer 2 connectivity between hosts for specific applications as well as a technique for interconnecting data centers (DCI). Nevertheless, VxLAN itself only provides the datacenter overlay encapsulation, so an overlay routing protocol such as BGP-EVPN is needed. Ethernet Virtual Private Network (EVPN) is a feature that provides a scalable, interoperable end-to-end control-plane solution for VXLAN tunnels over L3 infrastructure using BGP. But what about the underlay routing protocol to get from VTEP to VTEP? This post will cover and discuss the protocols most supported by networking vendors: OSPF, IS-IS, iBGP and eBGP.

Underlay protocols

Intra-domain routing protocols

OSPFv2 is a well understood protocol across network engineers, however it’s biggest limitation is that it is IPv4 only. OSPFv3 was developed to support IPv6 routes and later extended to support IPv4 routes as well. OSPF, however, was rejected by most web-scale operators because ofits lack of multiprotocol support. IS-IS, just like OSPF, is a link-state protocol, but IS-IS is a far better regarded protocol that can route both IPv4 and IPv6 stacks, being the single-protocol handling a common reason for enterprises and service providers to deploy. However, good IS-IS implementations are few, limiting the administrator’s choices. Furthermore, many operators felt that a link-state protocol was inherently unsuited for arichly connected networks such as the Clos topology. Link-state routing protocols rely on shared topology database flooded between participating nodes (routers). The whole link state domain is a single trust zone – a single node collapsing can bring down the whole domain. This is the reason why this protocols are not reccommended for mission critical network infrastructure.

Inter-domain routing protocols

BGP then appeared to bring features and behaviours the other two protocols didn't: BGP is mature, powers the internet, is simpler to understand, offers better vendor interoperability, allows per-hop traffic engineering, event propagation more constrained, speechless and supports multiprotocols (IPv4, IPv6, MPLS, VPNs,...). Although BGP was mainly used by service provicers and BGP could not be used inside the data center in the same way, Microsoft’s Azure conducted the BGP adaptation to the data center, since in the service provider network, stability is preferred over rapid notification of changes, but in the data center network, operators want routing updates to be as fast as possible. This routing design allows to build robust and scalable IP fabric, which provides plain L3 connectivity between connected endpoints. Furthermore, the use of External BGP (EBGP) has shown (RFC 7938) to be well suited as a stand-alone routing protocol for data center applications, and today is the most chosen underlay routing protocol to use with VxLAN EVPN in data centers.

Jaume Campeny and Alba Massa