Fabric Extend - Over Private WAN and Internet Simultaneously

[Ppenev]

Hello Extreme Community,

I am new to the Fabric Connect architecture/paradigm and I’m currently learning it by building an extensive lab with licensed physical equipment in a physical data center.

I’m in need of a little configuration guidance with regard to Fabric Extend and the extension of the SPBM fabric across L3 networks.

I am able to create an FE tunnel over an IP-VPN WAN between a VSP4900 at a Main site and a remote XA1440 appliance at a branch site.

I am also able to create an FE tunnel with IPSec over the Internet between an XA1440 at the Main site and the same remote XA1440 appliance.

What I can’t figure out is how to configure both at the same time - I am limited by only being able to configure a single ISIS ip-tunnel-source-address on the VSPs.

On each side, I am using a separate VRF to isolate the interfaces which connect to the L3 transport networks, and a I’m using a brouter port for the tunnel on both sides. The red brouter ports are sitting in a DMZ on each firewall with public IP addresses.

On the diagram below, I have full IP connectivity with some static routes inside the Fabric Extend VRF. Inside the VRF I am able to ping out to the Internet and at the same time I can ping across the WAN to the main site.

I can build an ISIS adjacency between the purple dots, and then when I change the ip-tunnel-source-address to the public IP I can get and ISIS adjacency between the red dots.

Remote XA:

***************

#
# ISIS CONFIGURATION
#

router isis
sys-name "Remote_XA"
ip-source-address 10.0.0.11
ip-tunnel-source-address 10.13.10.26 vrf fab_extend
is-type l1
system-id 0050.0000.aaaa
manual-area 10.01
exit
router isis enable

#
# LOGICAL ISIS CONFIGURATION
#

logical-intf isis 1 dest-ip 10.10.10.2 mtu 1950 name "Tunnel_to_LAK"
isis
isis spbm 1

isis enable
exit
logical-intf isis 2 dest-ip 168.168.210.250 mtu 1950 name "Tunnel_to_LAK_ipsec"
isis
isis spbm 1
isis enable
exit



#
# PORT CONFIGURATION - PHASE II
#

interface GigabitEthernet 1/1
name "L3 link to WAN cloud"
no shutdown
vrf fab_extend

brouter port 1/1 vlan 802 subnet 10.13.10.26/255.255.255.248 mac-offset 0

no spanning-tree mstp  force-port-state enable
exit
interface GigabitEthernet 1/2
name "L3 link to the Internet"
no shutdown
vrf fab_extend
brouter port 1/2 vlan 803 subnet 167.167.208.4/255.255.255.0 mac-offset 1

no spanning-tree mstp  force-port-state enable
exit




Remote_XA:1#show isis interface

========================================================================================================================
                                                    ISIS Interfaces
========================================================================================================================
IFIDX             TYPE    LEVEL     OP-STATE  ADM-STATE  ADJ    UP-ADJ  SPBM-L1-METRIC   OP-SPBM-L1-METRIC   ORIGIN
------------------------------------------------------------------------------------------------------------------------
Tunnel_to_LAK     pt-pt   Level 1   UP        UP         1      1       20000            20000               CONFIG
Tunnel_to_LAK_i~  pt-pt   Level 1   UP        UP         0      0       20000            20000               CONFIG

--------------------------------------------------------------------------------
 2 out of 2 Total Num of ISIS interfaces
--------------------------------------------------------------------------------


Remote_XA:1#show isis adjacencies

====================================================================================================
                                ISIS Adjacencies
====================================================================================================
INTERFACE         L STATE        UPTIME PRI HOLDTIME SYSID             HOST-NAME
----------------------------------------------------------------------------------------------------
Tunnel_to_LAK     1 UP         04:50:35 127       22 0048.0000.00c1    FC_LAK_Core1

--------------------------------------------------------------------------------
 1 out of 1 interfaces have formed an adjacency
--------------------------------------------------------------------------------

How can I have 2 permanent ISIS adjacencies to the Campus Fabric Connect environment across the two circuits?

Should I not  use public IP addresses on the IPSec tunnel endpoints and instead use the same tunnel source address as the VXLAN tunnel but NAT it at the firewalls?

Also, documentation mentions that the XA1400 appliances are hardened - do you know any details on that? What is mean by “hardened” in this context?

I am trying to achieve the scenario on the very left in the following screenshot:

Your thoughts are appreciated,

Thank you!

[Ppenev]

Thank you very much for your answer, Ludovico!

I have seen the concepts you are referring to in documentation, so I can understand what you are suggesting. (sorry for not mentioning anything about the code versions I use on the appliances, but I am indeed using VOSS 8.3.0.0)

If I have the necessary hardware resources at the remote branch site, then would you say that right now it is a better solution to implement the scenario shown in the middle of the screenshot below?

This way, at the remote branch I will have one VSP connected to one ISP circuit and another VSP connected to another ISP circuit, with an ISIS adjacency between the VSPs. If either of the WAN circuits was to go down, I would be relying on Fabric Connect to select the available path to the main site.

Thanks again for the valuable input!

[Ludovico_Steven · Accepted solution]

The ability for the same XA (VSP with FIGW) to connect to multiple providers is currently a bit limited and we are expecting improvements to this in the VOSS 8.3.1.0 release later this year.

However, in your specific case it is already doable as you have one IPsec tunnel and a native VXLAN tunnel. You need to use decoupled mode on the Remote XA IPsec config, using this command which was added to the XA in release 8.2:

8.2.0.0 SW	Fabric Extend ability to specify a separate global IPsec tunnel source address (ipsec tunnel-source-address) from the FE IP Tunnel source IP

That will allow your Remote XA to originate all its IPsec traffic via a different IPsec IP source address from the ISIS FE tunnel source IP (ip-tunnel-source-address). You can make this ipsec tunnel-source-address the Brouter port you have facing the Internet Firewall.

And of course, now on your HQ XA you will need to set:

8.2.0.0 SW	Fabric Extend ability to specify a logical ISIS interface IPsec tunnel destination (ipsec tunnel-dest-ip) other than the FE tunnel destination IP

...under the logical-isis interface to match that IPsec termination IP which is no longer the ip-tunnel-source-address of that same logical-isis interface.

However note that traffic arriving to the Remote XA, and destined to the new above IPsec tunnel-source-address, once the IPsec encapsulation is removed, there is still an inner VXLAN encapsulation which will still have as destination IP the ISIS FE ip-tunnel-source-address. So it is fine to make this the Brouter port facing the WAN circuit, provided that you use the same VRF for both WAN and Internet circuits (you have no choice anyway, we only support 1 FE enabled VRF), so that the VXLAN tunnel which was IPsec encapsulated can reach it. But if you use a Brouter port, and you lose the link to the WAN router, then both your WAN and Internet circuits will go down which is probably not what you want. You could make the ip-tunnel-source-address a CLIP IP on the XA, but now the snag is that the WAN provider will need an IP route to reach that CLIP. Or else you could make the ip-tunnel-source-address a VLAN IP (supported on XA since 8.1.5.0 and on VOSS VSPs since 8.3.0.0) and then assign that VLAN not only to the WAN port, but also to another bogus port you expect will always be up or at least be up as long as the Internet connection is available (for example the port connecting to your Internet Firewall !). But in this case, note that we don’t support an IP enabled VLAN to be assigned to a Brouter port. So simply make the Internet facing ipsec tunnel-source-address a VLAN IP as well.

Hope you can digest this!

[Ppenev]

Thank you, Miguel-Angel, for your response! I will have to take some time to think of what you’re suggesting and I will report back once I try it.

[Miguel-Angel_RO]

Ppenev,

Based on what you shared I would ad the following to your config.

REMOTE:
#

# PORT CONFIGURATION - PHASE II

interface GigabitEthernet 1/1

tx-flow-control

no shutdown

exit

#

# ISIS CONFIGURATION

router isis
ip tcp adjust-mss mss 1150

exit

#

# LOGICAL ISIS CONFIGURATION

# HERE THE MTU is adjusted to 1400 to cope with the ISP MTU

# Pay attention to the destination tunnel IP (local) and destination IPSEC IP (public)

logical-intf isis 1 dest-ip 168.168.210.250 mtu 1400 name "Tunnel_to_LAK_ipsec"

auth-key ***WhateverIPSEC_Key_HERE***

ipsec remote-nat-ip 167.167.208.1

ipsec

exit

HQ

#

# PORT CONFIGURATION - PHASE II

interface GigabitEthernet 1/1

tx-flow-control

exit

#

# IP CONFIGURATION -  GlobalRouter

# Here you have to add an explicit route to the private remote XA IP pointing to your default GW. This point seems strange but it is mandatory “by design”.

router vrf  fab_extend

ip route 0.0.0.0 0.0.0.0 #HERE THE DEFAULT GW# weight 1 preference 130

ip route 10.13.10.26 255.255.255.255 #HERE THE DEFAULT GW# weight 1 preference 5

#

# ISIS CONFIGURATION

router isis

ip-source-address 10.22.255.33

ip-tunnel-source-address 192.168.101.33

ip tcp adjust-mss mss 1150

exit

router isis enable

#

# LOGICAL ISIS CONFIGURATION

logical-intf isis 59 dest-ip 192.168.1.63 mtu 1500 name "GNP_CORE-1"

isis

isis spbm 1

isis enable

auth-key ***WhateverIPSEC_Key_HERE***

ipsec responder-only

ipsec

exit

In case of issues, please shared the following on all the switches:

show run modu isis

show ip route static

Mig