Booting after System Failure

  • 0
  • 1
  • Problem
  • Updated 2 years ago
  • Solved
  • (Edited)
Hi Guys,
Had an unexpected switch reboot. Checking the logs cannot find any particular reason. Any ideas?

ExtremeXOS 15.5.3.4  patch1-5

16-08-2016 08:37:44 <EPM.system_stable> System is stable. Change to warm reset mode
16-08-2016 08:37:44 <ACL.Policy.bind> Policy:bind:edge-ports-policy:vlan:*:port:1:11:
16-08-2016 08:37:44 <ACL.Policy.bind> Policy:bind:edge-ports-policy:vlan:*:port:1:10:
16-08-2016 08:37:44 <ACL.Policy.bind> Policy:bind:edge-ports-policy:vlan:*:port:1:9:
16-08-2016 08:37:44 <ACL.Policy.bind> Policy:bind:edge-ports-policy:vlan:*:port:1:8:
16-08-2016 08:37:44 <ACL.Policy.bind> Policy:bind:edge-ports-policy:vlan:*:port:1:7:
16-08-2016 08:37:44 <ACL.Policy.bind> Policy:bind:edge-ports-policy:vlan:*:port:1:6:
16-08-2016 08:37:44 <ACL.Policy.bind> Policy:bind:edge-ports-policy:vlan:*:port:1:5:
16-08-2016 08:37:44 <ACL.Policy.bind> Policy:bind:edge-ports-policy:vlan:*:port:1:4:
16-08-2016 08:37:44 <ACL.Policy.bind> Policy:bind:edge-ports-policy:vlan:*:port:1:3:
16-08-2016 08:37:44 <ACL.Policy.bind> Policy:bind:edge-ports-policy:vlan:*:port:1:2:
16-08-2016 08:37:44 <ACL.Policy.bind> Policy:bind:edge-ports-policy:vlan:*:port:1:1:
16-08-2016 08:37:03 <pm.config.loaded> Loaded Policy: edge-ports-policy number of entries 1
16-08-2016 08:37:03 <pm.config.openingFile> Loading policy edge-ports-policy from file /config/edge-ports-policy.pol
16-08-2016 08:36:38 <DM.Notice> Slot-5 being Powered ON
16-08-2016 08:36:38 <DM.Notice> Slot-4 being Powered ON
16-08-2016 08:36:38 <DM.Notice> Slot-3 being Powered ON
16-08-2016 08:36:38 <DM.Notice> Slot-2 being Powered ON
16-08-2016 08:36:35 <EPM.wdg_enable> Watchdog enabled
16-08-2016 08:36:24 <DM.Notice> Node State[3] = MASTER
16-08-2016 08:36:21 <SNMP.Subagent.MstrRestrt> snmpMaster process has been restarted.
16-08-2016 08:36:13 <SNMP.Master.InitDone> snmpMaster initialization complete
16-08-2016 08:36:12 <telnetd.info> **** telnetd started *****
16-08-2016 08:36:11 <tftpd.info> **** tftpd started *****
16-08-2016 08:36:11 <DOSProt.Init> DOS protect application started successfully
16-08-2016 08:36:08 <DM.Notice> Node State[2] = STANDBY
16-08-2016 08:36:08 <DM.Info> Node INIT DONE ....
16-08-2016 08:36:07 <HAL.Card.Info> Module in Slot-4 is inserted
16-08-2016 08:36:07 <HAL.Card.Info> Module in Slot-2 is inserted
16-08-2016 08:36:07 <HAL.Port.Info> Stacking port 4:2 link up at 10Gbps.
16-08-2016 08:36:07 <HAL.Port.Info> Stacking port 4:1 link up at 10Gbps.
16-08-2016 08:36:07 <HAL.Port.Info> Stacking port 2:2 link up at 10Gbps.
16-08-2016 08:36:07 <HAL.Port.Info> Stacking port 2:1 link up at 10Gbps.
16-08-2016 08:36:05 <HAL.Card.Info> Module in Slot-5 is inserted
16-08-2016 08:36:05 <HAL.Card.Info> Module in Slot-3 is inserted
16-08-2016 08:36:05 <DM.Notice> Slot-1 being Powered ON
16-08-2016 08:36:05 <DM.Notice> Node State[1] = INIT
16-08-2016 08:36:04 <HAL.Sys.Info> Hal initialization done.
16-08-2016 08:36:04 <HAL.Card.Info> Module in Slot-1 is inserted
16-08-2016 08:36:03 <HAL.Sys.Notice> Module in fan slot 8 is removed
16-08-2016 08:36:03 <HAL.Sys.Notice> Module in fan slot 7 is removed
16-08-2016 08:36:03 <HAL.Sys.Notice> Module in fan slot 6 is removed
16-08-2016 08:36:03 <HAL.Sys.Notice> Module in fan slot 5 is removed
16-08-2016 08:36:03 <HAL.Sys.Notice> Module in fan slot 4 is removed
16-08-2016 08:36:03 <HAL.Sys.Notice> Module in fan slot 3 is removed
16-08-2016 08:36:03 <HAL.Sys.Notice> Module in fan slot 2 is removed
16-08-2016 08:36:03 <HAL.Sys.Notice> Module in fan slot 1 is inserted
16-08-2016 08:36:03 <HAL.Port.Info> Stacking port 5:2 link up at 10Gbps.
16-08-2016 08:36:03 <HAL.Port.Info> Stacking port 5:1 link up at 10Gbps.
16-08-2016 08:36:03 <HAL.Port.Info> Stacking port 3:2 link up at 10Gbps.
16-08-2016 08:36:03 <HAL.Port.Info> Stacking port 3:1 link up at 10Gbps.
16-08-2016 08:36:03 <HAL.Port.Info> Stacking port 1:2 link up at 10Gbps.
16-08-2016 08:36:03 <HAL.Port.Info> Stacking port 1:1 link up at 10Gbps.
16-08-2016 08:36:03 <SNMP.Subagent.InitDone> snmpSubagent initialization complete
16-08-2016 08:36:03 <HAL.Sys.Info> Starting hal initialization ....
16-08-2016 08:36:02 <nl.init> Network Login framework has been initialized
16-08-2016 08:35:53 <telnetd.info> telnetd listening on port 23

16-08-2016 08:35:45 <DM.Notice> DM started
16-08-2016 08:35:45 <NM.StrtProc> The Node Manager (NM) has started processing.
16-08-2016 08:35:43 <EPM.start> EPM Started
16-08-2016 08:35:43 <EPM.UnexpctRebootDtect> Booting after System Failure.
16-08-2016 08:35:42 <EPM.wd_warm_reset> Changing to watchdog warm reset mode
16-08-2016 08:20:24 <Kern.IPv4Mc.Warning> IPv4 multicast entry not added.  Hardware L3 Table full. (Logged at most once per hour.)
16-08-2016 07:36:16 <Kern.IPv4Mc.Warning> IPv4 multicast entry not added.  Hardware L3 Table full. (Logged at most once per hour.)
16-08-2016 07:34:13 <Kern.IPv4Mc.Warning> IPv4 multicast entry not added.  Hardware L3 Table full. (Logged at most once per hour.)
16-08-2016 07:33:46 <Kern.IPv4Mc.Warning> IPv4 multicast entry not added.  Hardware L3 Table full. (Logged at most once per hour.)
16-08-2016 07:32:23 <Kern.IPv4Mc.Warning> IPv4 multicast entry not added.  Hardware L3 Table full. (Logged at most once per hour.)
16-08-2016 07:17:18 <Kern.IPv4Mc.Warning> IPv4 multicast entry not added.  Hardware L3 Table full. (Logged at most once per hour.)
16-08-2016 06:27:45 <Kern.IPv4Mc.Warning> IPv4 multicast entry not added.  Hardware L3 Table full. (Logged at most once per hour.)
16-08-2016 06:25:42 <Kern.IPv4Mc.Warning> IPv4 multicast entry not added.  Hardware L3 Table full. (Logged at most once per hour.)
16-08-2016 06:25:15 <Kern.IPv4Mc.Warning> IPv4 multicast entry not added.  Hardware L3 Table full. (Logged at most once per hour.)
16-08-2016 06:23:52 <Kern.IPv4Mc.Warning> IPv4 multicast entry not added.  Hardware L3 Table full. (Logged at most once per hour.)
16-08-2016 06:09:54 <Kern.IPv4Mc.Warning> IPv4 multicast entry not added.  Hardware L3 Table full. (Logged at most once per hour.)
16-08-2016 05:04:27 <Kern.IPv4Mc.Warning> IPv4 multicast entry not added.  Hardware L3 Table full. (Logged at most once per hour.)
16-08-2016 05:02:23 <Kern.IPv4Mc.Warning> IPv4 multicast entry not added.  Hardware L3 Table full. (Logged at most once per hour.)
16-08-2016 05:01:57 <Kern.IPv4Mc.Warning> IPv4 multicast entry not added.  Hardware L3 Table full. (Logged at most once per hour.)
16-08-2016 05:01:05 <Kern.IPv4Mc.Warning> IPv4 multicast entry not added.  Hardware L3 Table full. (Logged at most once per hour.)
16-08-2016 05:00:33 <Kern.IPv4Mc.Warning> IPv4 multicast entry not added.  Hardware L3 Table full. (Logged at most once per hour.)
16-08-2016 04:02:46 <Kern.IPv4Mc.Warning> IPv4 multicast entry not added.  Hardware L3 Table full. (Logged at most once per hour.)


Thank you,
Mykhaylo
Photo of Transporter

Transporter

  • 222 Points 100 badge 2x thumb

Posted 2 years ago

  • 0
  • 1
Photo of Ron Huygens

Ron Huygens, Employee

  • 2,998 Points 2k badge 2x thumb
Hello Mykhaylo,

For these kind of issues it is better to open a GTAC case for better trouble-shooting.

Ron
Photo of Transporter

Transporter

  • 222 Points 100 badge 2x thumb
Hi Ron,

Ok I have raised it through the support portal.

Thanks,
Mykhaylo
Photo of Transporter

Transporter

  • 222 Points 100 badge 2x thumb
GTAC update:
-------------------------------------------------------------------------------------------------------------------
ln the L3 Hardware table, the multicast entries are reaching the hardware limit.
 
In this case you can choose one of the two options: 

1. If you are not using multicast traffic you can disable IGMP snooping. If you disable it then it floods the multicast traffic like broadcast. 

2. Change the forwarding mode to group-vlan mode. 
The command is the following: 

configure forwarding ipmc lookup-key group-vlan 
configure forwarding ipmc compression group-table 
https://gtacknowledge.extremenetworks.com/articles/How_To/Multicast-Entry-not-Added-Hardware-Table-Full 

I would recommend the second option. After that the Hardware L3 Table full logs should not appear. 
--------------------------------------------------------------------------------------------------------------------
Photo of Grosjean, Stephane

Grosjean, Stephane, Employee

  • 12,816 Points 10k badge 2x thumb
For 1) you should also make sure you don't have an IP address on the vlan.
For 2) changing the lookup key is the best approach, indeed. Several options are opened to you that depend on your traffic type. Going (*,G) might help, or not. In a pure L2 environment I would consider the mac mode or the mixed one. Check user guide for guidances and limitations.

What is your platform?

In a peak usage, what sh iproute reserved-entries stat returns?
Photo of Transporter

Transporter

  • 222 Points 100 badge 2x thumb
Hello Grosjean,

Slot-1 LR-RM224-ES-2.7 # sh iproute reserved-entries stat                      |-----In HW Route Table-----|   |--In HW L3 Hash Table--|
                       # Used Routes   # IPv4 Hosts   IPv4   IPv4  IPv6   IPv4
Slot  Type              IPv4   IPv6    Local Remote   Local  Rem.  Local  MCast
----  --------------- ------- ------  ------ ------   -----  ----- -----  -----
1     X440-48p              3      0       1      0       0      0     0    207
2     X440-48p              3      0       1      0       0      0     0    207
3     X440-48p              3      0       1      0       0      0     0    207
4     X440-48p              3      0       1      0       0      0     0    207
5     X440-48p              3      0       1      0       0      0     0    207
6                           -      -       -      -       -      -     -      -
7                           -      -       -      -       -      -     -      -
8                           -      -       -      -       -      -     -      -

Thx,
Mykhaylo
Photo of Grosjean, Stephane

Grosjean, Stephane, Employee

  • 12,816 Points 10k badge 2x thumb
L3 Multicast HW table full is typical on x440. Definitely you need to change the lookup key, that will solve that issue in the logs.
Photo of Transporter

Transporter

  • 222 Points 100 badge 2x thumb
Thanks Grosjean. So second suggested solution will help us.