topic Re: Default gateway on VSP not working after reboot of default gateway in ExtremeSwitching (VSP/Fabric Engine)

Default gateway on VSP not working after reboot of default gateway

BRMS — Mon, 08 Jun 2020 17:45:09 GMT

We are using an SPBM-Cloud of 4 VSP 8600 as our Backbone. Two of them are connected to a layer 2 transport net in which the firewall is used as default gateway. last week we had a power shortage and a few weeks ago i rebooted the firewall at night. both times the VSP stopped using the firewall as gateway. clients that tried to ping something behind the firewall got an “time to life exceeded” error. The VSPs itself were able to ping devices behind the firewall.

By using different VRFs on the VSPs we are creating different security domains. all other VRFs didn’t suffer from that problem although they get routed by the same firewall, albeit another IP.

the solution to the problem was deleting the route and recreating it.

this is the route we are using:

ip route 0.0.0.0 0.0.0.0 172.28.2.1 weight 1 preference 5

show ip route
************************************************************************************
       Command Execution Time: Mon Jun 08 12:42:44 2020 CEST
************************************************************************************
=====================================================================================================
                                       IP Route - GlobalRouter
=====================================================================================================
                                                     NH                      INTER
DST             MASK            NEXT                 VRF/ISID         COST   FACE     PROT AGE TYPE PRF
-----------------------------------------------------------------------------------------------------
0.0.0.0         0.0.0.0         172.28.2.1           GlobalRouter     1      135      STAT 0   IB   5

this is the routing table on one of the VSPs that is not directly connected to the firewall:

show ip route
************************************************************************************
       Command Execution Time: Mon Jun 08 12:40:27 2020 CEST
************************************************************************************
=====================================================================================================
                                       IP Route - GlobalRouter
=====================================================================================================
                                                     NH                      INTER
DST             MASK            NEXT                 VRF/ISID         COST   FACE     PROT AGE TYPE PRF
-----------------------------------------------------------------------------------------------------
0.0.0.0         0.0.0.0         pik                  GlobalRouter     10     4051     ISIS 0   IBSE 7
0.0.0.0         0.0.0.0         kreuz                GlobalRouter     10     4051     ISIS 0   IBSE 7
0.0.0.0         0.0.0.0         pik                  GlobalRouter     10     4052     ISIS 0   IBSE 7
0.0.0.0         0.0.0.0         kreuz                GlobalRouter     10     4052     ISIS 0   IBSE 7

What could possibly be the reason for this strange behavior?

Re: Default gateway on VSP not working after reboot of default gateway

Martin_Sebek — Mon, 08 Jun 2020 18:32:47 GMT

Hello,

what version of VOSS are you using on VSP 8600? Do you use VRRP on VSPs in transport network? I’ve seen broken routing in GRT prior to version 6.2.0.3.

Re: Default gateway on VSP not working after reboot of default gateway

BRMS — Mon, 08 Jun 2020 19:15:19 GMT

im using the most recent one: 6.3.4.0

the firewall is connected via smlt with 2 of the VSPs. vrrp is used for ip redundancy:

this is the vlan config of both connected VSPs:

VSP1:

vlan create 135 name "tr_firewall" type port-mstprstp 0 
vlan mlt 135 16
vlan mlt 135 109
vlan mlt 135 110
vlan mlt 135 111
vlan mlt 135 112
vlan mlt 135 113
vlan mlt 135 114    
vlan mlt 135 115
vlan members 135 2/4,7/1-7/7 portmember
vlan i-sid 135 10135
interface Vlan 135
ip address 172.28.2.4 255.255.255.240 53 
ip vrrp version 3 
ip vrrp address 2 172.28.2.2
ip vrrp 2 backup-master enable
ip vrrp 2 enable
exit

show ip vrrp address 
************************************************************************************
		Command Execution Time: Mon Jun 08 14:14:10 2020 CEST
************************************************************************************

====================================================================================================
                            VRRP Info - GlobalRouter
====================================================================================================

VRRP ID  P/V      IP              MAC                STATE    CONTROL  PRIO  ADV VERSION 
----------------------------------------------------------------------------------------------------
[...]
2        135      172.28.2.2      00:00:5e:00:01:02  Backup   Enabled  100   1   3       

2 out of 2 Total Num of VRRP Address Entries displayed.


VRRP ID  P/V      MASTER          UP TIME               HLD DWN  CRITICAL IP(ENABLED)  VERSION
----------------------------------------------------------------------------------------------------
[...]
2        135      172.28.2.5      7 day(s), 01:11:12    0        0.0.0.0         (No)  3       

2 out of 2 Total Num of VRRP Address Entries displayed.

VSP2:

vlan create 135 name "tr_firewall" type port-mstprstp 0 
vlan mlt 135 16
vlan mlt 135 109
vlan mlt 135 110
vlan mlt 135 111
vlan mlt 135 112
vlan mlt 135 113
vlan mlt 135 114
vlan mlt 135 115
vlan members 135 2/4,7/1-7/7 portmember
vlan i-sid 135 10135
interface Vlan 135
ip address 172.28.2.5 255.255.255.240 52 
ip vrrp version 3 
ip vrrp address 2 172.28.2.2
ip vrrp 2 backup-master enable
ip vrrp 2 priority 200
ip vrrp 2 enable
exit

show ip vrrp address 
************************************************************************************
		Command Execution Time: Mon Jun 08 14:14:10 2020 CEST
************************************************************************************

====================================================================================================
                            VRRP Info - GlobalRouter
====================================================================================================

VRRP ID  P/V      IP              MAC                STATE    CONTROL  PRIO  ADV VERSION 
----------------------------------------------------------------------------------------------------
[...]
2        135      172.28.2.2      00:00:5e:00:01:02  Master   Enabled  200   1   3       

2 out of 2 Total Num of VRRP Address Entries displayed.


VRRP ID  P/V      MASTER          UP TIME               HLD DWN  CRITICAL IP(ENABLED)  VERSION
----------------------------------------------------------------------------------------------------
[...]
2        135      172.28.2.5      7 day(s), 01:34:20    0        0.0.0.0         (No)  3       

2 out of 2 Total Num of VRRP Address Entries displayed.

Re: Default gateway on VSP not working after reboot of default gateway

Martin_Sebek — Tue, 09 Jun 2020 07:18:58 GMT

I don’t have an experience with 6.3.4.0. Running 6.3.3.0 everywhere and I haven’t seen such a behavior. Configuration looks fine from what you’ve showed.

I would recommend you to open a case. It looks like the issue is pretty easy to reproduce so it should not be a problem for Extreme to look onto it. I am interested in further information about this issue as we are using similar setup. Would you please keep us updated?

Re: Default gateway on VSP not working after reboot of default gateway

BRMS — Tue, 09 Jun 2020 21:15:17 GMT

thx for the info, ill keep you updated. btw. the problem was the same with version 6.3.3.0, after the first occurrence i updated the systems in the hopes this will solve the problem, which it unfortunately didn’t.

Re: Default gateway on VSP not working after reboot of default gateway

Chi — Fri, 16 Oct 2020 23:49:52 GMT

Is the firewall pointing to 172.28.2.2 as its next hop for the networks in question?

Shot in the dark, do the routes match on all four VSP8600s? Going to need to do ISIS redistribution of static and direct routes between the four of them.

Re: Default gateway on VSP not working after reboot of default gateway

schuert — Tue, 10 Nov 2020 21:59:48 GMT

isis redistribution is enabled. both connected VSP have the firewall as default router and have rsmlt enabled. the other 2 VSP know of the default route via route redistribution.

since after the fw-uplink drops for 2 seconds the routing doesn’t work anymore for connected clients, but the VSP are still able to use the default gateway to reach systems behind the fw our partner guesses, that this could be a bug in the hardware routing table.

since the cli get routed in software and is able to use the gw, but the hardware routing table used by connected clients is not, this would be fitting. recreating the default route or resetting the gigabitethernet interfaces fixes the problem, which also causes an update of the hardware routing table.

as far as i remember the error appeared after we updated to version 6.3.4. at the moment we are using 6.3.5 with the same problems.

EDIT: btw. this is my personal account, im the OP.