topic RE: Booting after System Failure in ExtremeSwitching (EXOS/Switch Engine)

Booting after System Failure

Transporter — Tue, 16 Aug 2016 14:20:00 GMT

Hi Guys,
Had an unexpected switch reboot. Checking the logs cannot find any particular reason. Any ideas?

ExtremeXOS 15.5.3.4 patch1-5

16-08-2016 08:37:44 System is stable. Change to warm reset mode
16-08-2016 08:37:44 Policy:bind:edge-ports-policy:vlan:*:port:1:11:
16-08-2016 08:37:44 Policy:bind:edge-ports-policy:vlan:*:port:1:10:
16-08-2016 08:37:44 Policy:bind:edge-ports-policy:vlan:*:port:1:9:
16-08-2016 08:37:44 Policy:bind:edge-ports-policy:vlan:*:port:1:8:
16-08-2016 08:37:44 Policy:bind:edge-ports-policy:vlan:*:port:1:7:
16-08-2016 08:37:44 Policy:bind:edge-ports-policy:vlan:*:port:1:6:
16-08-2016 08:37:44 Policy:bind:edge-ports-policy:vlan:*:port:1:5:
16-08-2016 08:37:44 Policy:bind:edge-ports-policy:vlan:*:port:1:4:
16-08-2016 08:37:44 Policy:bind:edge-ports-policy:vlan:*:port:1:3:
16-08-2016 08:37:44 Policy:bind:edge-ports-policy:vlan:*:port:1:2:
16-08-2016 08:37:44 Policy:bind:edge-ports-policy:vlan:*:port:1:1:
16-08-2016 08:37:03 Loaded Policy: edge-ports-policy number of entries 1
16-08-2016 08:37:03 Loading policy edge-ports-policy from file /config/edge-ports-policy.pol
16-08-2016 08:36:38 Slot-5 being Powered ON
16-08-2016 08:36:38 Slot-4 being Powered ON
16-08-2016 08:36:38 Slot-3 being Powered ON
16-08-2016 08:36:38 Slot-2 being Powered ON
16-08-2016 08:36:35 Watchdog enabled
16-08-2016 08:36:24 Node State[3] = MASTER
16-08-2016 08:36:21 snmpMaster process has been restarted.
16-08-2016 08:36:13 snmpMaster initialization complete
16-08-2016 08:36:12 **** telnetd started *****
16-08-2016 08:36:11 **** tftpd started *****
16-08-2016 08:36:11 DOS protect application started successfully
16-08-2016 08:36:08 Node State[2] = STANDBY
16-08-2016 08:36:08 Node INIT DONE ....
16-08-2016 08:36:07 Module in Slot-4 is inserted
16-08-2016 08:36:07 Module in Slot-2 is inserted
16-08-2016 08:36:07 Stacking port 4:2 link up at 10Gbps.
16-08-2016 08:36:07 Stacking port 4:1 link up at 10Gbps.
16-08-2016 08:36:07 Stacking port 2:2 link up at 10Gbps.
16-08-2016 08:36:07 Stacking port 2:1 link up at 10Gbps.
16-08-2016 08:36:05 Module in Slot-5 is inserted
16-08-2016 08:36:05 Module in Slot-3 is inserted
16-08-2016 08:36:05 Slot-1 being Powered ON
16-08-2016 08:36:05 Node State[1] = INIT
16-08-2016 08:36:04 Hal initialization done.
16-08-2016 08:36:04 Module in Slot-1 is inserted
16-08-2016 08:36:03 Module in fan slot 8 is removed
16-08-2016 08:36:03 Module in fan slot 7 is removed
16-08-2016 08:36:03 Module in fan slot 6 is removed
16-08-2016 08:36:03 Module in fan slot 5 is removed
16-08-2016 08:36:03 Module in fan slot 4 is removed
16-08-2016 08:36:03 Module in fan slot 3 is removed
16-08-2016 08:36:03 Module in fan slot 2 is removed
16-08-2016 08:36:03 Module in fan slot 1 is inserted
16-08-2016 08:36:03 Stacking port 5:2 link up at 10Gbps.
16-08-2016 08:36:03 Stacking port 5:1 link up at 10Gbps.
16-08-2016 08:36:03 Stacking port 3:2 link up at 10Gbps.
16-08-2016 08:36:03 Stacking port 3:1 link up at 10Gbps.
16-08-2016 08:36:03 Stacking port 1:2 link up at 10Gbps.
16-08-2016 08:36:03 Stacking port 1:1 link up at 10Gbps.
16-08-2016 08:36:03 snmpSubagent initialization complete
16-08-2016 08:36:03 Starting hal initialization ....
16-08-2016 08:36:02 Network Login framework has been initialized
16-08-2016 08:35:53 telnetd listening on port 23

16-08-2016 08:35:45 DM started
16-08-2016 08:35:45 The Node Manager (NM) has started processing.
16-08-2016 08:35:43 EPM Started
16-08-2016 08:35:43 Booting after System Failure.
16-08-2016 08:35:42 Changing to watchdog warm reset mode
16-08-2016 08:20:24 IPv4 multicast entry not added. Hardware L3 Table full. (Logged at most once per hour.)
16-08-2016 07:36:16 IPv4 multicast entry not added. Hardware L3 Table full. (Logged at most once per hour.)
16-08-2016 07:34:13 IPv4 multicast entry not added. Hardware L3 Table full. (Logged at most once per hour.)
16-08-2016 07:33:46 IPv4 multicast entry not added. Hardware L3 Table full. (Logged at most once per hour.)
16-08-2016 07:32:23 IPv4 multicast entry not added. Hardware L3 Table full. (Logged at most once per hour.)
16-08-2016 07:17:18 IPv4 multicast entry not added. Hardware L3 Table full. (Logged at most once per hour.)
16-08-2016 06:27:45 IPv4 multicast entry not added. Hardware L3 Table full. (Logged at most once per hour.)
16-08-2016 06:25:42 IPv4 multicast entry not added. Hardware L3 Table full. (Logged at most once per hour.)
16-08-2016 06:25:15 IPv4 multicast entry not added. Hardware L3 Table full. (Logged at most once per hour.)
16-08-2016 06:23:52 IPv4 multicast entry not added. Hardware L3 Table full. (Logged at most once per hour.)
16-08-2016 06:09:54 IPv4 multicast entry not added. Hardware L3 Table full. (Logged at most once per hour.)
16-08-2016 05:04:27 IPv4 multicast entry not added. Hardware L3 Table full. (Logged at most once per hour.)
16-08-2016 05:02:23 IPv4 multicast entry not added. Hardware L3 Table full. (Logged at most once per hour.)
16-08-2016 05:01:57 IPv4 multicast entry not added. Hardware L3 Table full. (Logged at most once per hour.)
16-08-2016 05:01:05 IPv4 multicast entry not added. Hardware L3 Table full. (Logged at most once per hour.)
16-08-2016 05:00:33 IPv4 multicast entry not added. Hardware L3 Table full. (Logged at most once per hour.)
16-08-2016 04:02:46 IPv4 multicast entry not added. Hardware L3 Table full. (Logged at most once per hour.)

Thank you,
Mykhaylo

RE: Booting after System Failure

Ron_Huygens — Tue, 16 Aug 2016 14:56:00 GMT

Hello Mykhaylo,

For these kind of issues it is better to open a GTAC case for better trouble-shooting.

Ron

RE: Booting after System Failure

Transporter — Tue, 16 Aug 2016 16:43:00 GMT

Hi Ron,

Ok I have raised it through the support portal.

Thanks,
Mykhaylo

RE: Booting after System Failure

Transporter — Thu, 18 Aug 2016 15:54:00 GMT

GTAC update:
-------------------------------------------------------------------------------------------------------------------
ln the L3 Hardware table, the multicast entries are reaching the hardware limit.

In this case you can choose one of the two options:

1. If you are not using multicast traffic you can disable IGMP snooping. If you disable it then it floods the multicast traffic like broadcast.

2. Change the forwarding mode to group-vlan mode.
The command is the following:

configure forwarding ipmc lookup-key group-vlan
configure forwarding ipmc compression group-table
https://gtacknowledge.extremenetworks.com/articles/How_To/Multicast-Entry-not-Added-Hardware-Table-Full

I would recommend the second option. After that the Hardware L3 Table full logs should not appear.
--------------------------------------------------------------------------------------------------------------------

RE: Booting after System Failure

Stephane_Grosj1 — Thu, 18 Aug 2016 15:54:00 GMT

For 1) you should also make sure you don't have an IP address on the vlan. For 2) changing the lookup key is the best approach, indeed. Several options are opened to you that depend on your traffic type. Going (*,G) might help, or not. In a pure L2 environment I would consider the mac mode or the mixed one. Check user guide for guidances and limitations. What is your platform? In a peak usage, what sh iproute reserved-entries stat returns?

RE: Booting after System Failure

Transporter — Thu, 18 Aug 2016 15:54:00 GMT

Hello Grosjean,

Slot-1 LR-RM224-ES-2.7 # sh iproute reserved-entries stat |-----In HW Route Table-----| |--In HW L3 Hash Table--|
# Used Routes # IPv4 Hosts IPv4 IPv4 IPv6 IPv4
Slot Type IPv4 IPv6 Local Remote Local Rem. Local MCast
---- --------------- ------- ------ ------ ------ ----- ----- ----- -----
1 X440-48p 3 0 1 0 0 0 0 207
2 X440-48p 3 0 1 0 0 0 0 207
3 X440-48p 3 0 1 0 0 0 0 207
4 X440-48p 3 0 1 0 0 0 0 207
5 X440-48p 3 0 1 0 0 0 0 207
6 - - - - - - - -
7 - - - - - - - -
8 - - - - - - - -

Thx,
Mykhaylo

RE: Booting after System Failure

Stephane_Grosj1 — Thu, 18 Aug 2016 15:54:00 GMT

L3 Multicast HW table full is typical on x440. Definitely you need to change the lookup key, that will solve that issue in the logs.

RE: Booting after System Failure

Transporter — Thu, 18 Aug 2016 15:54:00 GMT

Thanks Grosjean. So second suggested solution will help us.