topic RE: mpls L3VPN between Cisco and Extreme Networks XOS devices in ExtremeSwitching (EXOS/Switch Engine)

mpls L3VPN between Cisco and Extreme Networks XOS devices

Stefano_Dall_Os — Thu, 30 Mar 2017 22:06:00 GMT

Hi everybody,
I'm trying to get L3VPN mpls working between Extreme Networks x460g2 and various cisco devices (3600, ASR920, 7600, 9000), and actually I'm stuck ...
Has anyone ever been able to do it?

I'll try to explain what I've done with some pictures and text information ...

Here are the L1 and L2/L3 schemes of version 1 of my lab ...

On each switch/router we have 2 loopback interfaces/vlans:
- 1 for OSPF 172.18.0.x/32
- 1 for iBGP 172.18.128.x/32
The «x» refers to the numeric ID of each switch/router, with the only exception of RFI1, where:
- OSPF loopback is 172.18.0.3/32
- iBGP loopback is 172.18.128.1/32
All switches are in the same OSPF area 172.18.128.217, and is the BGP AS 172. RFI1 is the RR for the BGP part, and the ONLY neighbor for each switch/router.

All ospf interfaces are PTP.
BGP and OSPF seems to work fine as soon as we DON’T enable MPLS.
LDP protocol seems to work well between the two vendors.
We created 2 VPN-VRF on every switch/router:
- vr-acme with RD 172:10 ad route-target 172:10 in both RX and TX, with a binded loopback interface 3.3.3.x/32
- vr-mgt_ool_104999 with RD 172:104999 and route-target 172:104999 in both RX ad TX, with a binded loopback interface 4.4.4.x/32

From my point of view, the main «suspect» is something in the routing part.
We changed the iBGP route priority in the extreme devices, to be similar to the Cisco administrative distance
configure iproute priority ibgp 4000
I still have doubts on «where» to put the priority of the MPLS.
I tried the default value, before iBGP or after iBGP, and the result is pretty close the same:
as soon as we enable the MPLS routing stuff, things start to work NOT in the way we expected/wanted.

Step1:
- We added the 2 loopback vlans and the ospf PTP vlan in the mpls and LDP «process».
- We enabled «mpls protocol ldp» and «mpls» itself
At this point, LDP starts to work, and we start to see some MPLS stuff, but the main goal, that is to see routing information on the two separate
VRF, is still not reached (we don’t see anything in the specific VRF routing table, as expected ... mpls routing is STILL not enabled)

Step2:
- We enable the MPLS routing
• enable iproute mpls-next-hop
• enable iproute mpls-next-hop vr vr-acme
• enable iproute mpls-next-hop vr vr-mgt_ool_104999
At this point, for a while (iBGP timeout), I see what I want to see in the VRF routing table (actually just the loopback interfaces binded to each VRF), but after the iBGP timeout, everything disappears.
The cause seems to be the fact that as soon as I enable the MPLS routing, I loose the reachability of the iBGP loopback interface, and from there I loose the iBPG neighborship.
And here is the MOST interesting part: the ISSUE is NOT everywhere, but just from a device
(and from that one, nothing works, like in a chain)
The «guilty device» is the FIRST cisco switch/router, no matter which model it is
(we tried to «switch» between cisco models, but nothing changed).

To be more specific, if we look at «version 1» of the test, if we try to ping from RFI1 using the iBGP loopback interface as source, and the iBGP
loopback interface of each other switch/router as the destination, we have:
- RFI1 can ping 217
- RFI1 can ping 216
- RFI1 CANNOT ping 213
- RFI1 CANNOT ping 214
- RFI1 CANNOT ping 215
Moreover: IF the chain is ONLY of extreme switches, everything works perfectly (still using RFI1, a cisco device, as RR ... same configuration ...)
Even more, just because RFI1 is a REAL production router, for a while I used a smaller set of devices.
Just take the same scheme of «version 1», remove RFI1, and take 217 as its replacement
(so 217 is the RR for iBGP, and all other router just have it as a neighbor).
In this way, everything works perfectly.

Here are pictures for version 2 of the same lab ... same results ...

RE: mpls L3VPN between Cisco and Extreme Networks XOS devices

andreas1 — Fri, 31 Mar 2017 00:20:00 GMT

Does Extreme support MBGP for VPNv4 familie ?

RE: mpls L3VPN between Cisco and Extreme Networks XOS devices

Stephane_Grosj1 — Fri, 31 Mar 2017 00:20:00 GMT

yes

RE: mpls L3VPN between Cisco and Extreme Networks XOS devices

Stefano_Dall_Os — Fri, 31 Mar 2017 00:20:00 GMT

from what I know, yes ...

RE: mpls L3VPN between Cisco and Extreme Networks XOS devices

Stephane_Grosj1 — Fri, 31 Mar 2017 14:35:00 GMT

I'm a bit puzzled with that part:

"Step2:
- We enable the MPLS routing
• enable iproute mpls-next-hop
• enable iproute mpls-next-hop vr vr-acme
• enable iproute mpls-next-hop vr vr-mgt_ool_104999"

MPLS can only be in a single VR. So I don't get that part of the config.

A typical config for L3VPN is, on a PE:

enable jumbo-frame ports all
create vlan lo0
create vlan vl101 tag 101

enable loopback-mode vlan lo0
configure vl101 add ports 2 tagged

configure lo0 ipaddress 172.16.0.2/32
configure vl101 ipaddress 10.1.1.5/30
enable ipforwarding

configure ospf routerid 172.16.0.2
configure ospf add lo0 area 0.0.0.0 passive
configure ospf add vl101 area 0.0.0.0 link-type point-to-point
enable ospf

configure mpls lsr-id 172.16.0.2
configure mpls ldp advertise direct lsr-id
configure mpls add lo0
configure mpls add vl101
enable mpls lo0
enable mpls vl101
enable mpls ldp lo0
enable mpls ldp vl101
enable mpls protocol ldp
enable mpls
enable iproute mpls-next-hop

configure bgp AS-number 65000
configure bgp routerid 172.16.0.2
create bgp neighbor 172.16.0.4 remote-AS-number 65000
configure bgp neighbor 172.16.0.4 source-interface ipaddress 172.16.0.2
configure bgp neighbor 172.16.0.4 next-hop-self
configure bgp neighbor 172.16.0.4 address-family vpnv4 next-hop-self
enable bgp neighbor 172.16.0.4 capability vpnv4
enable bgp neighbor 172.16.0.4
enable bgp

# the VPN VRF
create vr "vpn-a" type vpn-vrf vr "VR-Default"
configure vr VR-Default delete ports 1
configure vr vpn-a add ports 1

# vl100 is the interconnection with the CE
create vlan vl100 vr vpn-a tag 100
configure vl100 add ports 1 tagged
configure vl100 ipaddress 10.1.1.2/30
enable ipforwarding vlan vl100

# if you are using BGP for PE/CE
configure vr vpn-a add protocol bgp

configure vr vpn-a rd 172.16.0.2:100
configure vr vpn-a route-target both add 65000:100

# still assuming BGP for CE/PE
virtual-router vpn-a
configure bgp AS-number 65000
configure bgp routerid 172.16.0.2
create bgp neighbor 10.1.1.1 remote-AS-number 65100
enable bgp neighbor 10.1.1.1
enable bgp

# redistribution CE / PE
enable bgp export vr vpn-a direct address-family vpnv4
enable bgp export vr vpn-a bgp address-family vpnv4

# redistribution of remote routes
virtual-router vpn-a
enable bgp export remote-vpn

RE: mpls L3VPN between Cisco and Extreme Networks XOS devices

Stefano_Dall_Os — Sat, 01 Apr 2017 03:01:00 GMT

Hi,
and thanks a lot for the answer!

a couple of things ...
I'm checking the differences between your config and mine ...

you didn't enable the
"enable iproute mpls-next-hop"
on VRF vpn-a ...
you said it's only possible to enable mpls on just ONE vr ...
and I know it, but I thought since vpn-a is a VRF with vr-default as a "father" vr,
I thought I had to enable MPLS on all children VRFs ...

about these commands

# still assuming BGP for CE/PE
virtual-router vpn-a
configure bgp AS-number 65000
configure bgp routerid 172.16.0.2
create bgp neighbor 10.1.1.1 remote-AS-number 65100
enable bgp neighbor 10.1.1.1
enable bgp

# redistribution CE / PE
enable bgp export vr vpn-a direct address-family vpnv4
enable bgp export vr vpn-a bgp address-family vpnv4

# redistribution of remote routes
virtual-router vpn-a
enable bgp export remote-vpn

these are for the CE/PE connection, right?

anyway, I have to test what you suggested in my lab with the costumer ... let's see if it works ...

but have you ever had the chance to make Cisco and Extreme devices work together with mpls L3VPN ?!

thanks a lot

best regards

Stefano

RE: mpls L3VPN between Cisco and Extreme Networks XOS devices

Stephane_Grosj1 — Sat, 01 Apr 2017 03:01:00 GMT

Hi, yes I have customers doing L3VPN with Cisco for years.

RE: mpls L3VPN between Cisco and Extreme Networks XOS devices

Stefano_Dall_Os — Sat, 01 Apr 2017 03:01:00 GMT

and you use the configuration you sent, don't you? ... I mean,
part of it ...

do you change anything in the ip route priority stuff?! ... since extreme and cisco have different "administrative distance"/"route priority" default configuration ...

thanks again ...

RE: mpls L3VPN between Cisco and Extreme Networks XOS devices

Stephane_Grosj1 — Sat, 01 Apr 2017 03:01:00 GMT

Hi,

No, I usually try not to change the protocol preference. As for the config, for a PE, yes, it's usually similar to that. Of course you need to adapt. I didn't past typical P router config, but that's just MPLS, I assume you're fine with that.

RE: mpls L3VPN between Cisco and Extreme Networks XOS devices

Stefano_Dall_Os — Sat, 01 Apr 2017 03:01:00 GMT

looking at the Cisco and Extreme configuration I posted below, do you see anything wrong? ... anything that can prevent L3VPN to work as expected between the 2 vendors? ... do you have a typical CISCO config? ... thanks a lot ... 🙂

RE: mpls L3VPN between Cisco and Extreme Networks XOS devices

Stefano_Dall_Os — Wed, 05 Apr 2017 17:52:00 GMT

mmm ...
I'm still having issues ...
I tried to follow the suggestions,
but still it's NOT working as expected ...

this is the CISCO configuration "template"

!
ip vrf acme
rd 172:10
route-target export 172:10
route-target import 172:10
!
### THE OSPF LOOPBACK ###
interface Loopback0
ip address 172.18.0.213 255.255.255.255
!
### THE LOOPBACK related to one VRF ###
interface Loopback10
ip vrf forwarding acme
ip address 3.3.3.213 255.255.255.255
!
### THE BGP LOOPBACK ###
interface Loopback99
description IP riferimento Neighbor Mpls
ip address 172.18.128.213 255.255.255.255
!
interface GigabitEthernet0/1
no switchport
ip address 10.151.217.10 255.255.255.252
ip ospf network point-to-point
mpls ip
!
interface GigabitEthernet0/2
no switchport
ip address 10.151.217.17 255.255.255.252
ip ospf network point-to-point
mpls ip
!
router ospf 172
router-id 172.18.0.213
auto-cost reference-bandwidth 40000
area 172.18.128.217 nssa no-summary
passive-interface default
no passive-interface GigabitEthernet0/1
no passive-interface GigabitEthernet0/2
network 10.151.217.8 0.0.0.3 area 172.18.128.217
network 10.151.217.16 0.0.0.3 area 172.18.128.217
network 172.18.0.213 0.0.0.0 area 172.18.128.217
network 172.18.128.213 0.0.0.0 area 172.18.128.217
!
router bgp 172
bgp router-id 172.18.128.213
bgp log-neighbor-changes
no bgp default ipv4-unicast
neighbor 172.18.128.1 remote-as 172
neighbor 172.18.128.1 update-source Loopback99
!
address-family ipv4
exit-address-family
!
address-family vpnv4
neighbor 172.18.128.1 activate
neighbor 172.18.128.1 send-community both
exit-address-family
!
address-family ipv4 vrf acme
redistribute connected
exit-address-family
!
mpls ldp router-id Loopback99 force
!

this is the EXTREME NETWORKS template

#
# Module vlan configuration.
#
configure vlan default delete ports all
configure vr VR-Default delete ports 1-34
create vr "vr-acme" type vpn-vrf vr "VR-Default"
create vr "vr-mgt_ool_104999" type vpn-vrf vr "VR-Default"
configure vlan default delete ports 1-34
enable jumbo-frame ports all
create vlan "vloop4000"
configure vlan vloop4000 tag 4000
enable loopback-mode vlan vloop4000
create vlan "vloop4009"
configure vlan vloop4009 tag 4009
enable loopback-mode vlan vloop4009
create vlan "vloop4010" vr vr-acme
configure vlan vloop4010 tag 4010
enable loopback-mode vlan vloop4010
create vlan "vloop4069" vr vr-mgt_ool_104999
configure vlan vloop4069 tag 4069
enable loopback-mode vlan vloop4069
create vlan "vp2p3001"
configure vlan vp2p3001 tag 3001
create vlan "vp2p3002"
configure vlan vp2p3002 tag 3002
configure vlan vp2p3001 add ports 1 untagged
configure vlan vp2p3002 add ports 2 untagged
configure vlan vloop4000 ipaddress 172.18.0.216 255.255.255.255
enable ipforwarding vlan vloop4000
configure vlan vloop4009 ipaddress 172.18.128.216 255.255.255.255
enable ipforwarding vlan vloop4009
configure vlan vp2p3001 ipaddress 10.151.217.6 255.255.255.252
enable ipforwarding vlan vp2p3001
configure vlan vp2p3002 ipaddress 10.151.217.9 255.255.255.252
enable ipforwarding vlan vp2p3002
configure vlan vloop4010 ipaddress 3.3.3.216 255.255.255.255
enable ipforwarding vlan vloop4010
configure vlan vloop4069 ipaddress 4.4.4.216 255.255.255.255
enable ipforwarding vlan vloop4069
configure vr vr-acme add protocol bgp
configure vr vr-mgt_ool_104999 add protocol bgp
configure vr vr-acme rd 172:10
configure vr vr-mgt_ool_104999 rd 172:104999
configure vr vr-acme route-target both add 172:10
configure vr vr-mgt_ool_104999 route-target both add 172:104999

#
# Module rtmgr configuration.
#
enable iproute mpls-next-hop
disable iproute ipv4 compression

#
# Module bgp configuration.
#
configure bgp AS-number 172
configure bgp routerid 172.18.128.216
enable bgp mpls-next-hop
enable bgp community format AS-number:number
create bgp neighbor 172.18.128.1 remote-AS-number 172
configure bgp neighbor 172.18.128.1 source-interface ipaddress 172.18.128.216
enable bgp neighbor 172.18.128.1
configure bgp neighbor 172.18.128.1 send-community both
configure bgp neighbor 172.18.128.1 next-hop-self
configure bgp neighbor 172.18.128.1 address-family vpnv4 next-hop-self
enable bgp neighbor 172.18.128.1 capability vpnv4
enable bgp export vr vr-acme direct address-family vpnv4
enable bgp export vr vr-acme bgp address-family vpnv4
enable bgp export vr vr-mgt_ool_104999 direct address-family vpnv4
enable bgp export vr vr-mgt_ool_104999 bgp address-family vpnv4
enable bgp

#
# Module bgp configuration on virtual router vr-acme.
#
virtual-router vr-acme
configure bgp AS-number 172
configure bgp routerid 172.18.128.216
enable bgp community format AS-number:number
enable bgp export remote-vpn address-family ipv4-unicast
enable bgp
virtual-router VR-Default

#
# Module bgp configuration on virtual router vr-mgt_ool_104999.
#
virtual-router vr-mgt_ool_104999
configure bgp AS-number 172
configure bgp routerid 172.18.128.216
enable bgp community format AS-number:number
enable bgp export remote-vpn address-family ipv4-unicast
enable bgp
virtual-router VR-Default

#
# Module mpls configuration.
#
configure mpls add vlan "vloop4000"
enable mpls vlan "vloop4000"
enable mpls ldp vlan "vloop4000"
configure mpls add vlan "vloop4009"
enable mpls vlan "vloop4009"
enable mpls ldp vlan "vloop4009"
configure mpls add vlan "vp2p3001"
enable mpls vlan "vp2p3001"
enable mpls ldp vlan "vp2p3001"
configure mpls add vlan "vp2p3002"
enable mpls vlan "vp2p3002"
enable mpls ldp vlan "vp2p3002"
configure mpls lsr-id 172.18.128.216
enable snmp traps mpls
enable mpls protocol ldp
enable mpls

#
# Module ospf configuration.
#
configure ospf routerid 172.18.0.216
enable ospf mpls-next-hop
configure ospf metric-table 10M 4000 100M 400 1G 40 10G 4 40G 1 100G 1
enable ospf
create ospf area 172.18.128.217
configure ospf area 172.18.128.217 nssa nosummary stub-default-cost 1000
configure ospf add vlan vloop4000 area 172.18.128.217 link-type point-to-point passive
configure ospf add vlan vloop4009 area 172.18.128.217 link-type point-to-point passive
configure ospf add vlan vp2p3001 area 172.18.128.217 link-type point-to-point
configure ospf add vlan vp2p3002 area 172.18.128.217 link-type point-to-point

I'm still having same issues ... with loss of BGP neighborship as soon as we "traverse" the first cisco device, so:
- 215 CANNOT ping
- 214 CANNOT ping
- 213 CANNOT ping

more over, leaving iproute priority default values, also 216 (the second extreme device) has the same issue ...
with

configure iproute priority mpls 3000
configure iproute priority ibgp 4000

at least 216 start to work again ...

thanks in advance

best regards

Stefano

RE: mpls L3VPN between Cisco and Extreme Networks XOS devices

Necheporenko__N — Wed, 05 Apr 2017 18:38:00 GMT

Hi,

Please check next one article - https://gtacknowledge.extremenetworks.com/articles/Solution/Ping-fails-over-L3VPN-tunnel-when-the-co...

BR,
Nikolay

RE: mpls L3VPN between Cisco and Extreme Networks XOS devices

Stefano_Dall_Os — Wed, 05 Apr 2017 20:05:00 GMT

Thanks Nikolay,
but I think that's not my case ...
it's not related to ARP ...

this is my output

VrId Gateway MAC Intf RCnt Flags
================================================================
3 003.003.003.216 00:00:00:00:00:00 0 1 ----L----
4 004.004.004.216 00:00:00:00:00:00 0 1 ----L----
2 010.151.217.005 00:04:96:98:FB:A8 3 4 RS-------
2 010.151.217.006 00:00:00:00:00:00 0 2 ----L----
2 010.151.217.009 00:00:00:00:00:00 0 2 ----L----
2 010.151.217.010 84:B8:02:69:3D:1F 4 1 RS-------
0 127.000.000.001 00:00:00:00:00:00 0 1 ----L----
2 172.018.000.216 00:00:00:00:00:00 0 1 ----L----
2 172.018.128.216 00:00:00:00:00:00 0 1 ----L----

and I don't see anything with the E flag ...

thanks again

best regards

Stefano

RE: mpls L3VPN between Cisco and Extreme Networks XOS devices

andreas1 — Thu, 06 Apr 2017 17:27:00 GMT

What is your MTU settings on cisco and extreme ?

RE: mpls L3VPN between Cisco and Extreme Networks XOS devices

Stefano_Dall_Os — Thu, 06 Apr 2017 17:42:00 GMT

I didn't touch the default values ... on both sides ...

RE: mpls L3VPN between Cisco and Extreme Networks XOS devices

Erik_Auerswald — Fri, 07 Apr 2017 19:24:00 GMT

Hi Stefano,

if I understand you correctly, moving the RR from the Cisco 7600 to an Extreme X460-G2 results in a working setup. Since the route reflector by default reflects the best route only, the route selection process might differ such that the Cisco router propagates (reflects) different routes than the Extreme switch. Perhaps even the VPNv4 routes are missing from the IOS RR.

Did you compare the advertised routes of the EXOS RR to those of the Cisco IOS RR?

Just an idea on how to proceed...

Thanks,
Erik

RE: mpls L3VPN between Cisco and Extreme Networks XOS devices

Stefano_Dall_Os — Fri, 07 Apr 2017 19:24:00 GMT

I tried ...
but still it works a little bit better, like at least I see routing information in the VRF,
but I cannot ping anything .. only from the 217, that is the first "extreme" hop ...

I still think there's something missing in the routing part ... it's like the 2 vendors does not speak the same identical language .... there seems to be a very slightly difference that I cannot find ...