EVPN NVO and Data Center Gateway with Nokia SR and Juniper MX – Part 1

In my previous blogs, I showed how to provision MPLS-based network services such as ePipe, VPLS and L3VPN (VPRN), which are common network services offered by carriers all over the world. One would note that the L2 epipe and VPLS, and the L3VPN network services have a very different control plane design. For the layer 2 network services, IP and MAC addresses are discovered via the age-old ARP protocol and data plane flooding. For the L3VPN network service, a dedicated control plane running MP-BGP (vpn-ipv4 and/or vpn-ipv6 addresses) is used by PEs to discover each VPN’s routes, route-targets (i.e., VPN memberships), route distinguisher etc… There is a clear control and data plane separation for MPLS L3VPN network service.

With MPLS network service deployment experience gained by carriers and equipment vendors in the past decade, a new set of L2 and L3 network services called Ethernet Virtual Private Network (EVPN) – RFC 7432 are developed in 2015 to supersede the MPLS based L2 and L3 network services. EVPN network services are well received by equipment vendors and customers due to its following advantages:

  • Unified control plane for L2 and L3 network services (e.g., L2 E-LAN, L2 E-TREE, L2 E-LINE, L3 VPN, L3 Overlay for Cloud and Data Center) based on MP-BGP (evpn address). The clear separation of control and data planes minimize L2 broadcast storm on data plane caused by network flooding for address discovery
  • Highly scalable network services
    • Use the time-tested and highly scalable BGP as the control plane protocol for MAC and IP addresses discovery
    • Support Mass Withdrawal to remove failed EVPN nodes and links as the next hop
  • All-active multi-home network service connection with per-flow redundancy and per-flow load-balancing
  • Choices of data planes to run EVPN network services such as:
    • EVPN-MPLS (RFC 7432)
    • EVPN-PBB (RFC-7623)
    • EVPN-NVO (Network Virtualization Overlay draft-ietf-bess-evpn-overlayLAN)

Out of the three EVPN data plane options, EVPN-NVO receives a lot of attentions as it fulfills many requirements outlined in RFC 7364 for a modern Data Center (DC) and Cloud architecture to:

  • Support a large number (e.g., 100,000s) of tenants in a DC
  • Isolated Unicast, and Broadcast, Unknown and Multicast (BUM) traffic per tenant
  • Extend L2 connectivity among Virtual Machines (VMs) of a tenant segment (subnet) across different Point of Deliveries (PODs) within a DC or across DCs
  • Allowing a VM to move or migrate between different PODs within a given L2 segment

In this article, I will discuss the modern SDN-based DC architecture and show how EVPN Network Virtualization Overlay (NVO) can be used to facilitate a DC tenant’s VMs to connect to the DC gateways for external network accesses such as VM inter-DC communication and VM floating IP Internet access.

In order to keep this article to a reasonable length for reading, I will divide this article into two parts. Part 1 (this article) covers all the introduction materials such as VXLAN, SDN, EVPN NVO, modern SDN-based DC architecture, and how EVPN NVO can be used to connect a tenant’s VMs to the DC gateways using Nokia SR. Part 2 of this article will cover Juniper MX EVPN NVO provisioning and interconnection of the two Nokia SR and Juniper MX DCs over an MPLS WAN.

Similar to my other articles, I will use GNS3 running on my home Windows 10 PC to demonstrate the EVPN NVO and inter-DC connection setup on Nokia SR and Juniper MX.

Introduction to VXLAN

There are two modern DC architectures namely Cloe (Leaf-Spin) and Overlay design. For this article, we will focus on DCs that are based on the Overlay design.

In an overlay DC design, layer 2 (L2) connectivity is provided among VMs running on different servers in a DC. L2 connectivity facilities VM move or migration from different PODs and it also offers lower switching cost than L3 routing. If a DC supported only Ethernet VLANs for tenants’ traffic separation, we could only support 4,094 tenants or VLAN IDs in a DC. This is clearly no sufficient for a modern DC that needs to support 100,000s tenants.

Virtual Extensible LAN (VXLAN) is based on the draft RFC, draft-mahalingam-dutt-dcops-vxlan. It is an overlay IP tunneling technology used to carry Ethernet traffic (Overlay) over any IP network (Underlay), and it is becoming the de facto standard for overlay data centers and networks as it supports uni-cast and multi-cast traffic with multi-path scalability through ECMP. Instead of using 12 bits (i.e., 4094) for VLAN IDs in Ethernet Dot1q, VXLAN uses a 24 bit (i.e., 16 millions) VXLAN Network Identifier (VNI) for DC tenant traffic separation. Conceptually, a DC tenant can have many VMs (i.e., Overlay IP addresses) located on many different servers (i.e., Underlay IP addresses) in a DC. As long as the VMs are tunneling using the same VNI, conceptually, they are on an isolated L2 LAN supporting the tenant’s Unicast, and Broadcast, Unknown, Multicast (BUM) traffic. Together with the DC Interconnect (DCI) that we will cover in Part 2 of this article, the virtual L2 LAN can be extended across different DCs for VM connectivity and migration etc….

We will come back to the Overlay and Underlay IP addresses when we discuss the GNS3 network setup.

Introduction to SDN

Software Defined Network (SDN) and Network Function Virtualization (NFV) are the two popular networking buzzwords found in networking conferences and publications in the past few years. There are many “principles” of SDN depending on whom you talk to. The separation of control and data planes is a common agreeable principle of SDN. The other key principles of SDN are:

  • Centralized vs. Distributed Network Control

Since the introduction of TCP/IP in the 70s, data network nodes (e.g., routers) and protocols were designed with distributed control in mind and the networks re-config themselves to bypass network failures. With this distributed network control, each network node has only partial information and status of the end-to-end network. Network engineers are then required to provision each network node with protocols to “stitch” the network together for the desired services, network capability and redundancy etc…. The distributed network control idea is being questioned by SDN that favors centralized network control. In SDN, there is a single entity in the system that understands the up-to-date and end-to-end network configuration and status, and network services can then be quickly deployed with the help of this centralized information without manually node-by-node CLI provisioning by network engineers

  • Programmable Network

With an up-to-date and end-to-end centralized view and status of a SDN network, programmable APIs can be offered by the centralized system in SDN for more intelligent network services. For example, an API create_path_max_bw (from_A, to_B, priority) can be offered by the centralized SDN system to create a network path between points A and B to fully utilize any available network bandwidth of the network at a given moment and dynamically adjust the paths and bandwidth with respect to other higher priority traffic, if necessary. This kind of network service can maximize the usages of network resources that distributed control protocols such as RSVP-TE cannot match

Introduction to SDN-Based Overlay Data Center

The principles of SDN can be applied to any networks such as DC, Metro, WAN etc… At this moment, DC and customer self-serve VPN networks are the first SDN networking products commonly available from many networking vendors. The traditional multi-tier Access, Aggregation and Core DC design is no longer suitable for new DCs that are proliferated with millions of VMs, Containers, NICs and East-West DC traffic. The following shows a modern SDN-based Overlay DC architecture to address the compute and network virtualization, and DC East-West traffic:

data center and SDN

Modern SDN-based Overlay Data Center

The Data Center Orchestration software (e.g., Openstack) has an end-to-end compute, network, authentication, images, status and control of the DC. Through the compute control messages, it can create, delete, modify, start and stop VMs and Containers running on any servers within its administrative domain (e.g., a DC).

Based on the DC tenant’s network topology specified at the DC orchestration software, VMs and/or Containers are created and their network configuration such as IP and MAC addresses, subnet, firewall, floating-IP, virtual ports, and QoS etc… are communicated to Network Virtualization Endpoints (NVEs) via Openflow and vendor specific network control messages. Some servers’ hypervisor support NVO operations and they can directly connect to the DC’s IP fabric. For legacy servers’ hypervisor that do not support NVO operations, they can be connected to NVO devices such as Top-of-Rack switches to the DC’s IP fabric.

GNS3 Overlay Data Center Setup

The following shows the network topology on GNS3 for two DCs interconnected via an MPLS WAN. Within each DC, a VPLS EVPN network is setup for each DC tenant or VNI to terminate its VM traffic to the DC gateways for inter-DC connections or floating IP Internet access.

Only the DC gateways and interconnection subsystems are modeled under GNS3 to illustrate the operations of EVPN NVO for connecting a DC tenant’s VMs to external networks and DC.

Data Center

SR-GW1, SR-GW2, and SR-NVE5 are Nokia Service Routers. MX-GW3, MX-GW4 and MX-NVE7 are Juniper MX routers. We will show how VMs connected to a NVE such as SR-NVE5 can join VNI 1000 to the DC gateways via a VPLS EVPN service. With EVPN NVO, the DC tenant now has an isolated L2 LAN (VNI 1000) for its unicast and BUM VMs’ traffic to the DC gateways.

Note that regular Nokia SR (e.g., SR-VNE5) and Juniper MX (e.g., MX-NVE7) routers are used in the above example for DC’s NVE operations. Both vendors offer specially designed NVO hardware that are more suitable for DC deployment such as higher Ethernet port count, 1 RU height etc…. The provisioning of these DC VNEs are via the SDN Controllers using Openflow messages instead of manual CLI provisioning as we are going to show in the later sections. Nevertheless, the manual CLI provisioning on SR-NVE5 and MX-NVE7 (Part 2 of the article) are still useful to show the underneath NVO operations of NVEs in a DC.

All vPCs such as VM22, VM44, VM55, and VM77 are VMs in the DCs for EVPN NVO testing purpose. SR-NVE5 and MX-NVE7 are Network Virtualization Endpoint (VNE) that offer VTEP tunneling for VM traffic.

Only one DC tenant is setup and it uses VNI 1000 in DC1 and VNI 2000 in DC2. In other words, the MPLS WAN config that we are going to cover in Part 2 of this article needs to do some VNI mapping between VNIs 1000 and 2000 over MPLS. In Part 2 of this article, VM migration over the EVPN NVO will be demonstrated such that VM55’s MAC and IP addresses can be migrated to say, VM22 to continuous its services while its server is under maintenance.

control plane

The following shows the logical diagram where VPLS EVPN 1000 service instances connects a tenant’s VMs to the DC gateways. There can be many VPLS EVPN instances at the DC gateways for DC tenants who need external network access for their VMs.

The following shows the physical setup for the VPLS EVPN 1000 (VxLAN 1000) to connect VMs to the DC Gateways to WAN.

EVPN NVO

The flexibility of overlay and underlay networking is shown in the above diagram. All NVEs (e.g., servers) and DC Gateways have underlay primary and backup network connections to the DC’s IP switching fabric for physical network connectivity. The underlay network topology is rather static. However, overlay network topology can be very flexiable and different. For example, in the above diagram, we setup VM-bl as the load balancer for VMa and VMb. All VMs need to go through VM-bl before they can reach the external network. While all VMs are on the same LAN (VxLAN 1000), different overlay subnets and Firewalls (FWs) can be used for traffic separation and security among the VMs. For example, VMa and VMb can use overlay IP subnets of 10.10.1.0/24 and 10.10.2.0/24 respectivley to VM-bl. FWs in front of VMa and VMb can be designed to allow only certain traffic such as HTTPS, MYSQL to go through. Other overlay network topologies such as hub-and-spoke, daisy-chain etc… are all possible over a given underlay network topology.

EVPN are applied to the Overlay DC in the following two areas in this article:

  • EVPN among NVEs and DC Gateways for VMs to access the DC gateways. One instance of VPLS EVPN is create for a tenant for a given VNI
  • EVPN between DCs for inter-DC VM communication

Introduction to EVPN Control Plane – MP-BGP (evpn address)

In EVPN, all VMs’ IP and MAC addresses are discovered via MP-BGP (evpn address) protocol. The following highlights MP-BGP (evpn address) Types 2 and 3 messages that are exchanged among NVEs for VM discovery and reachability. We will debug these MP-BGP (evpn address) messages in the later section:

  • Type 2 message – MAC/IP Advertisement route

Allows an end host’s IP and MAC addresses to be advertised within the EVPN Network Layer reachability information (NLRI)

  • Type 3 message – Inclusive Multicast Ethernet Tag route

Sets up a path for broadcast, unknown unicast, and multicast (BUM) traffic from a PE device to the remote PE device on a per-VLAN, per-ESI basis

Nokia SR EVPN NVO Setup

The following shows the provisioning steps for EVPN NVO operations on Nokia SR. A similar setup will be covered in Part 2 of the article for Juniper MX routers.

  1. Setup IGP on all NVEs such as TORs, SDN controllers, and DC gateways etc…
  2. Setup EVPN’s control plane protocol MP-BGP (evpn address) among the NVEs for VMs’ IP and MAC addresses discovery
  3. Setup EVPN’s data plane using VPLS network service with VXLAN encapsulation
  4. Verify the EVPN setup and confirm that VMs over the VPLS EVPN can communicate among each other using VXLAN encapsulation. Once the VMs’ traffic is at the DC gateways, it can be forwarded to other DCs over the DCI connection (Part 2 of this article)

Step 1 – IGP Setup

All NVEs such as TORs, SDN controllers and DC gateways need to have IGP setup for IP reachability. OSPF is used in this article. The following shows the OSPF setup for SR-GW2. Similar OSPF config are applied on SR-GW1 and SR-NVE5:

A:SR-GW2>config>router# info 
----------------------------------------------
#--------------------------------------------------
echo "IP Configuration"
#--------------------------------------------------
        interface "system"
            address 10.10.100.2/32
            no shutdown
        exit
        interface "toSR-GW1"
            address 10.1.2.2/27
            port 1/1/1
            no shutdown
        exit
        interface "toSR-NVE5"
            address 10.2.5.2/27
            port 1/1/2
            no shutdown
        exit
        autonomous-system 65501
#--------------------------------------------------
echo "OSPFv2 Configuration"
#--------------------------------------------------
        ospf 0                        
            area 0.0.0.0
                interface "system"
                    no shutdown
                exit
                interface "toSR-GW1"
                    no shutdown
                exit
                interface "toSR-NVE5"
                    no shutdown
                exit
            exit
            no shutdown
        exit

The following shows that the OSPF setup is successful among SR-GW1, SR-GW2, and SR-VNE5.

A:SR-GW2>config>router# show router ospf neighbor 

===============================================================================
Rtr Base OSPFv2 Instance 0 Neighbors
===============================================================================
Interface-Name                   Rtr Id          State      Pri  RetxQ   TTL
   Area-Id
-------------------------------------------------------------------------------
toSR-GW1                         10.10.100.1     Full       1    0       31
   0.0.0.0
toSR-NVE5                        10.10.100.5     Full       1    0       37
   0.0.0.0

 

Step 2 – EVPN’s Control Plane Setup with MP-BGP (EVPN)

The following shows the MP-BGP EVPN control plane configuration on SR-GW2. Similar configs are used on SR-GW1 and SR-NVE5.

A:SR-GW2>config>router>bgp# info 
----------------------------------------------
            vpn-apply-import
            vpn-apply-export
            group "nve-gw"
                family evpn
                type internal
                neighbor 10.10.100.1
                exit
                neighbor 10.10.100.5
                exit
            exit
            no shutdown

The above BGP config performs the following actions:

  • Advertise MAC addresses learned on SAPs, SDP bindings and conditional static MACs to the remote BGP peers
  • Ingress replication of Broadcast, Unknown unicast and Multicast (BUM) packets over VXLAN
  • MAC mobility and static-MAC protection as described in draft-ietf-l2vpnevpn, as well as MAC duplication detection

The following shows the successful BGP neighbor relationship among the DC gateways and NVEs. BGP Route Reflectors can be found to reduce the number of BGP neighbors in an actual DC deployment.

A:SR-GW2>config>router# show router bgp summary 
===============================================================================
 BGP Router ID:10.10.100.2      AS:65501       Local AS:65501      
===============================================================================
BGP Admin State         : Up          BGP Oper State              : Up
Total Peer Groups       : 1           Total Peers                 : 2         
Total VPN Peer Groups   : 0           Total VPN Peers             : 0         
Total BGP Paths         : 13          Total Path Memory           : 3320      
      
===============================================================================
BGP Summary
===============================================================================
Legend : D - Dynamic Neighbor
===============================================================================
Neighbor
Description
                   AS PktRcvd InQ  Up/Down   State|Rcv/Act/Sent (Addr Family)
                      PktSent OutQ
-------------------------------------------------------------------------------
10.10.100.1
               65501       80    0 00h38m40s 1/1/1 (Evpn)
                           81    0           
10.10.100.5
               65501       54    0 00h25m47s 1/1/1 (Evpn)
                           56    0     

 

Step 3 – EVPN’s Data Plane Setup for VXLAN

The following shows the EVPN NVO configuration using VXLAN encapsulation. The config supports VNI 1000 for a DC tenant. Other DC tenants will have a similar EVPN NVO configuration for their VNIs.

A:SR-GW2>config>router# /configure service 
A:SR-GW2>config>service# vpls 1000 
A:SR-GW2>config>service>vpls# info 
----------------------------------------------
            vxlan vni 1000 create
            exit
            bgp
                route-distinguisher 65501:2
                route-target export target:65501:1000 import target:65501:1000
            exit
            bgp-evpn
                vxlan
                    no shutdown
                exit
                mpls
                    shutdown
                exit
            exit
            stp
                shutdown
            exit
            sap 1/1/5:0 create
                no shutdown
            exit
            spoken-sdp X:Y create
  /* WAN connection to Internet or anther DC */
            no shutdown

sap 1/1/5:0 is only a testing interface for connecting a VM (e.g., VM22 for SR-GW2) for testing purpose. vxlan vni 1000 create setup a virtual overlay L2 LAN to connect all NVEs and the DC gateways for VNI 1000 for a tenant over the DC’s IP fabric.

Once the tenant’s VM traffic on VXLAN ID 1000 is terminated onto VPLS 1000, spoke-sdp X:Y of VPLS 1000 connects the tenant’s VM traffic to Internet or another DC. We will talk about DC’s WAN connection in more details in Part 2 of this article. At this moment, just consider spoke-sdp X:Y is the DC’s WAN connection placeholder for illustration purpose.

The network port MTU (in all the ports sending/receiving VXLAN packets) must be at least 50-bytes (54 if dot1q encapsulation is used) greater than the service MTU in order to accommodate the size of the VXLAN header. We will cover MTU setting in Part 2 of this article.

Step 4 – EVPN NVO Verification

To prove that EVPN NVO is setup correctly for SR-NVE5, SR-GW1 and SR-GW2, we connect VM55 and VM22 to SR-NVE5 and SR-GW2 respectively to ping each other over the VXLAN VNI 1000.

VM ping

As expected, VM55 and VM22 can ping each other over the VNI 1000.

VXLAN ping

The Wireshark trace captured on GNS3 shows the underlay addresses (e.g., 10.10.100.5 and 10.10.100.2) carrying the overlay addresses (192.168.100.55 and 192.168.100.22) and their payloads using VXLAN encapsulation for VNI 1000.

As discussed before, EVPN uses MP-BGP (evpn address) to learn IP and MAC addresses of VMs in a DC. The following debug commands on Nokia SR-GW2 shows the VMs’ MAC addresses exchanged between SR-NVE5 and SR-GW2 using MP-BPG EVPN Type 2 EVPN-MAC messages. MP-BGP EVPN Type 3 EVPN-Incl-mcas messages are also found for identifying BUM endpoints.

 
A:SR-GW2# config log log-id 6
A:SR-GW2# from debug to session
A:SR-GW2# debug router bgp packets

Note that in the below debug trace, VMs’ MAC addresses are:

  • VM55’s MAC address = 00:50:79:66:68:03
  • VM22’s MAC address = 00:50:79:66:68:00
18 2018/01/31 15:49:55.72 UTC MINOR: DEBUG #2001 Base Peer 1: 10.10.100.5
"Peer 1: 10.10.100.5: UPDATE
Peer 1: 10.10.100.5 - Received BGP UPDATE:
    Withdrawn Length = 0
    Total Path Attr Length = 88
    Flag: 0x90 Type: 14 Len: 44 Multiprotocol Reachable NLRI:
        Address Family EVPN
        NextHop len 4 NextHop 10.10.100.5
        Type: EVPN-MAC Len: 33 RD: 65501:1 ESI: ESI-0, tag: 0, mac len: 48 mac: 
00:50:79:66:68:03, IP len: 0, IP: NULL, label1: 1000 
    Flag: 0x40 Type: 1 Len: 1 Origin: 0
    Flag: 0x40 Type: 2 Len: 0 AS Path:
    Flag: 0x80 Type: 4 Len: 4 MED: 0
    Flag: 0x40 Type: 5 Len: 4 Local Preference: 100
    Flag: 0xc0 Type: 16 Len: 16 Extended Community:
        target:65501:1000
        bgp-tunnel-encap:VXLAN
"

19 2018/01/31 15:49:55.72 UTC MINOR: DEBUG #2001 Base Peer 1: 10.10.100.5
"Peer 1: 10.10.100.5: UPDATE
Peer 1: 10.10.100.5 - Received BGP UPDATE:
    Withdrawn Length = 0
    Total Path Attr Length = 84
    Flag: 0x90 Type: 14 Len: 28 Multiprotocol Reachable NLRI:
        Address Family EVPN
        NextHop len 4 NextHop 10.10.100.5
        Type: EVPN-Incl-mcast Len: 17 RD: 65501:1, tag: 0, orig_addr len: 32, or
ig_addr: 10.10.100.5
    Flag: 0x40 Type: 1 Len: 1 Origin: 0
    Flag: 0x40 Type: 2 Len: 0 AS Path:
    Flag: 0x80 Type: 4 Len: 4 MED: 0
    Flag: 0x40 Type: 5 Len: 4 Local Preference: 100
    Flag: 0xc0 Type: 16 Len: 16 Extended Community:
        target:65501:1000
        bgp-tunnel-encap:VXLAN
        Flag: 0xc0 Type: 22 Len: 9 PMSI:
        Tunnel-type Ingress Replication (6)
        Flags: (0x0)[Type: None BM: 0 U: 0 Leaf: not required]
        MPLS Label 1000
        Tunnel-Endpoint 10.10.100.5
"

22 2018/01/31 15:49:59.60 UTC MINOR: DEBUG #2001 Base Peer 1: 10.10.100.5
"Peer 1: 10.10.100.5: UPDATE
Peer 1: 10.10.100.5 - Send BGP UPDATE:
    Withdrawn Length = 0
    Total Path Attr Length = 88
    Flag: 0x90 Type: 14 Len: 44 Multiprotocol Reachable NLRI:
        Address Family EVPN
        NextHop len 4 NextHop 10.10.100.2
        Type: EVPN-MAC Len: 33 RD: 65501:2 ESI: ESI-0, tag: 0, mac len: 48 mac: 
00:50:79:66:68:00, IP len: 0, IP: NULL, label1: 1000 
    Flag: 0x40 Type: 1 Len: 1 Origin: 0
    Flag: 0x40 Type: 2 Len: 0 AS Path:
    Flag: 0x80 Type: 4 Len: 4 MED: 0
    Flag: 0x40 Type: 5 Len: 4 Local Preference: 100
    Flag: 0xc0 Type: 16 Len: 16 Extended Community:
        target:65501:1000
        bgp-tunnel-encap:VXLAN
"

23 2018/01/31 15:49:59.61 UTC MINOR: DEBUG #2001 Base Peer 1: 10.10.100.5
"Peer 1: 10.10.100.5: UPDATE
Peer 1: 10.10.100.5 - Send BGP UPDATE:
    Withdrawn Length = 0
    Total Path Attr Length = 84
    Flag: 0x90 Type: 14 Len: 28 Multiprotocol Reachable NLRI:
        Address Family EVPN
        NextHop len 4 NextHop 10.10.100.2
        Type: EVPN-Incl-mcast Len: 17 RD: 65501:2, tag: 0, orig_addr len: 32, or
ig_addr: 10.10.100.2
    Flag: 0x40 Type: 1 Len: 1 Origin: 0
    Flag: 0x40 Type: 2 Len: 0 AS Path:
    Flag: 0x80 Type: 4 Len: 4 MED: 0
    Flag: 0x40 Type: 5 Len: 4 Local Preference: 100
    Flag: 0xc0 Type: 16 Len: 16 Extended Community:
        target:65501:1000
        bgp-tunnel-encap:VXLAN
    Flag: 0xc0 Type: 22 Len: 9 PMSI:
        Tunnel-type Ingress Replication (6)
        Flags: (0x0)[Type: None BM: 0 U: 0 Leaf: not required]
        MPLS Label 1000
        Tunnel-Endpoint 10.10.100.2
"

The following shows the VPLS EVPN 1000’s fdb. The MAC addresses of VM22 and VM55 are shown.

A:SR-GW2# show service id 1000 fdb detail 

===============================================================================
Forwarding Database, Service 1000
===============================================================================
ServId    MAC               Source-Identifier        Type     Last Change
                                                     Age      
-------------------------------------------------------------------------------
1000      00:50:79:66:68:00 sap:1/1/5:0              L/0      01/30/18 02:43:01
1000      00:50:79:66:68:03 vxlan:                   Evpn     01/30/18 02:43:07
                            10.10.100.5:1000
-------------------------------------------------------------------------------
No. of MAC Entries: 2
-------------------------------------------------------------------------------
Legend:  L=Learned O=Oam P=Protected-MAC C=Conditional S=Static Lf=Leaf

The MAC addresses learned via MP-BGP (evpn) is shown below:

A:SR-GW2#  show router bgp routes evpn mac 
===============================================================================
 BGP Router ID:10.10.100.2      AS:65501       Local AS:65501      
===============================================================================
 Legend -
 Status codes  : u - used, s - suppressed, h - history, d - decayed, * - valid
                 l - leaked, x - stale, > - best, b - backup, p - purge
 Origin codes  : i - IGP, e - EGP, ? - incomplete

===============================================================================
BGP EVPN MAC Routes
===============================================================================
Flag  Route Dist.         MacAddr           ESI
      Tag                 Mac Mobility      Label1
                          Ip Address        
                          NextHop           
-------------------------------------------------------------------------------
u*>i  65501:1             00:50:79:66:68:03 ESI-0
      0                   Seq:0             VNI 1000
                          N/A
                          10.10.100.5

-------------------------------------------------------------------------------
Routes : 1                            

The following shows the VTEPs for VNI 1000 for the DC’s tenant:

A:SR-NVE5# show service id 1000 vxlan 
===============================================================================
Vxlan Src Vtep IP: N/A
===============================================================================
VPLS VXLAN, Ingress VXLAN Network Id: 1000
Creation Origin: manual
Assisted-Replication: none
RestProtSrcMacAct: none

===============================================================================
VPLS VXLAN service Network Specifics
===============================================================================
Ing Net QoS Policy : none                            Vxlan VNI Id     : 1000
Ingress FP QGrp    : (none)                          Ing FP QGrp Inst : (none)

===============================================================================
Egress VTEP, VNI
===============================================================================
VTEP Address                            Egress VNI  Num. MACs   Mcast Oper  L2
                                                                      State PBR
-------------------------------------------------------------------------------
10.10.100.1                             1000        0           BUM   Up    No
10.10.100.2                             1000        1           BUM   Up    No
-------------------------------------------------------------------------------
Number of Egress VTEP, VNI : 2

We just show that the tenant that is assigned by the DC administrator with VXLAN ID 1000 can use VPLS 1000 to terminate all it’s VMs’ traffic to the DC Gateways SR-GW1 and SR-GW2 and use spoke-sdp X:Y on VPLS 1000 to extend the VM traffic to Internet or another DC. We will discuss DC’s WAN connection in more details in Part 2 of this article.

Conclusion

We just show how to setup EVPN NVO on DC gateways using Nokia SRs so that a tenant’s VMs can have external network access via the DC gateways. VXLAN is used to encapsulate or tunnel VMs’ traffic for a given VNI (e.g., VNI 1000) to the DC gateways. EVPN control plane uses MP-BGP (evpn address) to discover VMs’ MAC addresses and BUM traffic remote endpoints.

The EVPN NVO configuration on the DC gateways such as SR-GW1 and SR-GW2 are usually setup manually like we just did in a production DC for security reason. In a production DC with 100,000s tenants, equipment vendors normally offer scripts to automate the EVPN NVO provisioning processes on the DC gateways.

In Part 2 of this article, we will complete EVPN NVO setup on Juniper MX routers and then interconnect the two DCs over an MPLS WAN using also EVPN (i.e. I-ES) to take advantage of EVPN’s unique features such as active-active redundancy, aliasing, and mass MAC withdrawal upon node and link failure. We will also show how a tenant’s VM on DC1 can migrate to DC2 for server maintenance. Stay tune.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s