Two switch X460G2 rebooted unexpectedly after a high CPU of process hal

earroyo — Wed, 08 May 2019 23:28:33 GMT

Good Morning, I had a problem this 06/05 near midnight with two switches that experimented high CPU after a BW test of 10G though a single 10G link. Im adjusting the topology,

The Logs were

Puerto Maldonado SW:

05/07/2019 00:00:47.29 Previous message repeated 5 additional times in the last 21 second(s)
05/07/2019 00:00:43.01 [VR 0x00000002] Peer 10.10.54.177 (0) has entered or left Established state, EST? 1, holdtime: 3, passive: 0
05/07/2019 00:00:41.47 MTU mismatch for DD Packet from neighbor 10.1.4.169 local MTU 9000 recvd MTU 9100.
05/07/2019 00:00:38.03 Port 16 link UP at speed 1 Gbps and full-duplex
05/07/2019 00:00:37.91 Port 16 link down
05/07/2019 00:00:37.47 MTU mismatch for DD Packet from neighbor 10.1.4.169 local MTU 9000 recvd MTU 9100.
05/07/2019 00:00:36.67 Changing the state of neighbor rtid 10.1.4.169 ipa 10.197.0.211 to state = EX_START due to two way event.
05/07/2019 00:00:34.97 Changing the state of neighbor rtid 10.1.4.169 ipa 10.197.0.211 to state = INIT due to hello received.
05/07/2019 00:00:34.97 Changing the state of neighbor rtid 10.1.4.169 ipa 0.0.0.0 to state = DOWN due to new neighbor.
05/07/2019 00:00:34.63 An attempt to process an ATG_I3_INET_ADDR_IND has failed due to local resource shortages or indication that the I3 join has lost some information for the request VR = 0x00000002 I3 join index = 1 Data lost flag value = 1 Local resource ret code = 0
05/07/2019 00:00:33.84 Port 52 link UP at speed 10 Gbps and full-duplex
05/07/2019 00:00:33.64 Port 51 link UP at speed 10 Gbps and full-duplex
05/07/2019 00:00:33.38 Port 50 link UP at speed 10 Gbps and full-duplex
05/07/2019 00:00:33.36 Port 49 link UP at speed 10 Gbps and full-duplex
05/07/2019 00:00:33.35 Port 21 link UP at speed 1 Gbps and full-duplex
05/07/2019 00:00:33.31 Input voltage to Internal PSU-2 is on. Output enabled.
05/07/2019 00:00:33.31 Internal PSU-2 is present.
05/07/2019 00:00:33.30 Input voltage to Internal PSU-1 is on. Output enabled.
05/07/2019 00:00:33.30 Internal PSU-1 is present.
05/07/2019 00:00:33.24 Port 17 link UP at speed 1 Gbps and full-duplex
05/07/2019 00:00:33.23 Port 16 link UP at speed 1 Gbps and full-duplex
05/07/2019 00:00:33.22 Media SF+_ER is inserted into Port 52
05/07/2019 00:00:33.22 Media SF+_LR is inserted into Port 51
05/07/2019 00:00:33.22 Media SF+_LR is inserted into Port 50
05/07/2019 00:00:33.22 Media SF+_LR is inserted into Port 49
05/07/2019 00:00:33.22 Media LX is inserted into Port 32
05/07/2019 00:00:33.22 Media LX100 is inserted into Port 31
05/07/2019 00:00:33.22 Media LX is inserted into Port 28
05/07/2019 00:00:33.22 Media LX is inserted into Port 26
05/07/2019 00:00:33.22 Media LX100 is inserted into Port 23
05/07/2019 00:00:33.22 Media LX100 is inserted into Port 22
05/07/2019 00:00:33.21 Media LX is inserted into Port 13
05/07/2019 00:00:33.21 Media LX is inserted into Port 12
05/07/2019 00:00:33.21 Media LX is inserted into Port 9
05/07/2019 00:00:33.17 Port 15 link UP at speed 1 Gbps and full-duplex
05/07/2019 00:00:33.05 Port 14 link UP at speed 1 Gbps and full-duplex
05/07/2019 00:00:32.89 Port 11 link UP at speed 1 Gbps and full-duplex
05/07/2019 00:00:32.88 Port 10 link UP at speed 1 Gbps and full-duplex
05/07/2019 00:00:32.71 Port 2 link UP at speed 1 Gbps and full-duplex
05/07/2019 00:00:32.62 Port 32 link UP at speed 1 Gbps and full-duplex
05/07/2019 00:00:32.62 Port 28 link UP at speed 1 Gbps and full-duplex
05/07/2019 00:00:32.61 Port 26 link UP at speed 1 Gbps and full-duplex
05/07/2019 00:00:32.60 Port 23 link UP at speed 1 Gbps and full-duplex
05/07/2019 00:00:32.60 Port 13 link UP at speed 1 Gbps and full-duplex
05/07/2019 00:00:32.59 Port 12 link UP at speed 1 Gbps and full-duplex
05/07/2019 00:00:32.59 Port 9 link UP at speed 1 Gbps and full-duplex
05/07/2019 00:00:32.01 Port Mgmt link UP at speed 1 Gbps and full-duplex
05/07/2019 00:00:32.01 Switch is operational
05/07/2019 00:00:26.06 snmpMaster initialization complete
05/07/2019 00:00:25.93 Loaded Policy: FROM-SBJ-TDP number of entries 1
05/07/2019 00:00:25.93 Loading policy FROM-SBJ-TDP from file /config/FROM-SBJ-TDP.pol
05/07/2019 00:00:24.33 NVRAM is full, old messages are overwritten.
05/07/2019 00:00:22.75 System is stable. Change to warm reset mode
05/07/2019 00:00:21.33 Msg from Master : Existing host key with fingerprint ec:e2:8e:65:e7:8b:13:14:80:74:92:79:77:f8:5b:55 [MD5] loaded successfully
05/07/2019 00:00:21.28 Msg from Master : Generating RSA-2048 public key
05/07/2019 00:00:21.27 Msg from Master : Loaded Private Key of size 1679 from System
05/07/2019 00:00:20.83 telnetd listening on port 2351
05/07/2019 00:00:20.82 Loaded Policy: telnet_in number of entries 1
05/07/2019 00:00:20.82 Loading policy telnet_in from file /config/telnet_in.pol
05/07/2019 00:00:19.89 The IP-MTU size 9216 for vlan v120 may be too large for jumbo-frame-size 9216. IP packets larger than 9194 may be lost.
05/07/2019 00:00:17.82 Watchdog enabled
05/07/2019 00:00:17.36 Node State[3] = OPERATIONAL
05/07/2019 00:00:16.22 Internal PSU-2 is disconnected.
05/07/2019 00:00:16.22 Internal PSU-1 is disconnected.
05/07/2019 00:00:14.39 DOS protect application started successfully
05/07/2019 00:00:13.93 snmpMaster process has been restarted.
05/07/2019 00:00:13.93 snmpSubagent initialization complete
05/07/2019 00:00:13.86 Node State[2] = STANDBY
05/07/2019 00:00:13.86 Node INIT DONE ....
05/07/2019 00:00:13.76 **** telnetd started *****
05/07/2019 00:00:13.75 **** tftpd started *****
05/07/2019 00:00:13.36 Node State[1] = INIT
05/07/2019 00:00:13.29 Network Login framework has been initialized
05/07/2019 00:00:13.20 Hal initialization done.
05/07/2019 00:00:12.63 Internal PSU-2 is powered off.
05/07/2019 00:00:12.63 Internal PSU-2 is present.
05/07/2019 00:00:12.63 Internal PSU-1 is powered off.
05/07/2019 00:00:12.63 Internal PSU-1 is present.
05/07/2019 00:00:12.55 Module in fan slot 1 is inserted
05/07/2019 00:00:12.37 Interface ID 2: name=exbcmpkt0 type=2 vlan=0 port=1
05/07/2019 00:00:04.09 Starting hal initialization ....
05/07/2019 00:00:02.85 The Event Management System logging server has started.
05/07/2019 00:00:02.84 The Node Manager (NM) has started processing.
05/07/2019 00:00:02.74 DM started
05/07/2019 00:00:02.58 EPM Started
05/07/2019 00:00:02.57 Booting after System Failure.
05/07/2019 00:00:01.39 Changing to watchdog warm reset mode

05/06/2019 23:14:10.85 CPU utilization monitor: process hal consumes 91 % CPU

Desaguadero Logs

05/07/2019 00:58:02.25 Port 51 link UP at speed 10 Gbps and full-duplex
05/07/2019 00:56:30.66 Save configuration failed due to 1 unresponsive application( fdb).
05/07/2019 00:56:11.98 Port 51 link down - Local fault
05/07/2019 00:35:54.63 Save configuration failed due to 1 unresponsive application( fdb).
05/07/2019 00:30:44.67 Save configuration failed due to 1 unresponsive application( fdb).
05/07/2019 00:20:34.00 Port 52 link UP at speed 10 Gbps and full-duplex
05/07/2019 00:20:33.94 Port 51 link UP at speed 10 Gbps and full-duplex
05/07/2019 00:20:33.49 Port 9 link UP at speed 1 Gbps and full-duplex
05/07/2019 00:20:33.49 Port 7 link UP at speed 1 Gbps and full-duplex
05/07/2019 00:20:33.44 Input voltage to Internal PSU-2 is on. Output enabled.
05/07/2019 00:20:33.44 Internal PSU-2 is present.
05/07/2019 00:20:33.44 Port 1 link UP at speed 1 Gbps and full-duplex
05/07/2019 00:20:33.42 Input voltage to Internal PSU-1 is on. Output enabled.
05/07/2019 00:20:33.42 Internal PSU-1 is present.
05/07/2019 00:20:33.39 Port 48 link UP at speed 1 Gbps and full-duplex
05/07/2019 00:20:33.39 Port 47 link UP at speed 1 Gbps and full-duplex
05/07/2019 00:20:33.34 Port 5 link UP at speed 1 Gbps and full-duplex
05/07/2019 00:20:33.24 Switch is operational
05/07/2019 00:20:30.97 Media SF+_LR is inserted into Port 52
05/07/2019 00:20:30.97 Media SF+_LR is inserted into Port 51
05/07/2019 00:20:30.97 Media SF+_LR is inserted into Port 50
05/07/2019 00:20:30.97 Media SF+_LR is inserted into Port 49
05/07/2019 00:20:30.97 Media LX is inserted into Port 48
05/07/2019 00:20:30.97 Media LX100 is inserted into Port 47
05/07/2019 00:20:30.97 Media LX is inserted into Port 9
05/07/2019 00:20:30.97 Media LX is inserted into Port 8
05/07/2019 00:20:30.97 Media LX is inserted into Port 7
05/07/2019 00:20:30.96 Media LX is inserted into Port 6
05/07/2019 00:20:30.96 Media LX is inserted into Port 5
05/07/2019 00:20:27.27 snmpMaster initialization complete
05/07/2019 00:20:26.07 NVRAM is full, old messages are overwritten.
05/07/2019 00:20:24.29 System is stable. Change to warm reset mode
05/07/2019 00:20:23.13 telnetd listening on port 2351
05/07/2019 00:20:23.13 Loaded Policy: telnet_in number of entries 1
05/07/2019 00:20:23.12 Loading policy telnet_in from file /config/telnet_in.pol
05/07/2019 00:20:23.12 Msg from Master : Existing host key with fingerprint 5d??18:1c:9d:3c:b4??f7:a3:ee:c6:03:6d:06:29 [MD5] loaded successfully
05/07/2019 00:20:23.08 Msg from Master : Generating RSA-2048 public key
05/07/2019 00:20:23.07 Msg from Master : Loaded Private Key of size 1675 from System

05/07/2019 00:20:21.83 The IP-MTU size 9216 for vlan v120 may be too large for jumbo-frame-size 9216. IP packets larger than 9194 may be lost.
05/07/2019 00:20:21.72 The IP-MTU size 9216 for vlan WAN-Desag-Ilave may be too large for jumbo-frame-size 9216. IP packets larger than 9194 may be lost.
05/07/2019 00:20:20.41 Watchdog enabled
05/07/2019 00:20:18.73 Node State[3] = OPERATIONAL
05/07/2019 00:20:17.42 Internal PSU-2 is disconnected.
05/07/2019 00:20:17.42 Internal PSU-1 is disconnected.
05/07/2019 00:20:15.36 **** telnetd started *****
05/07/2019 00:20:15.23 Node State[2] = STANDBY
05/07/2019 00:20:15.23 Node INIT DONE ....
05/07/2019 00:20:15.20 DOS protect application started successfully
05/07/2019 00:20:15.17 snmpMaster process has been restarted.
05/07/2019 00:20:15.16 snmpSubagent initialization complete
05/07/2019 00:20:15.03 **** tftpd started *****
05/07/2019 00:20:14.73 Node State[1] = INIT
05/07/2019 00:20:14.62 Network Login framework has been initialized
05/07/2019 00:20:14.41 Hal initialization done.
05/07/2019 00:20:13.83 Internal PSU-2 is powered off.
05/07/2019 00:20:13.83 Internal PSU-2 is present.
05/07/2019 00:20:13.83 Internal PSU-1 is powered off.
05/07/2019 00:20:13.83 Internal PSU-1 is present.
05/07/2019 00:20:13.75 Module in fan slot 1 is inserted
05/07/2019 00:20:13.58 Interface ID 2: name=exbcmpkt0 type=2 vlan=0 port=1
05/07/2019 00:20:05.25 Starting hal initialization ....
05/07/2019 00:20:04.20 The Event Management System logging server has started.
05/07/2019 00:20:04.14 The Node Manager (NM) has started processing.
05/07/2019 00:20:04.08 DM started
05/07/2019 00:20:03.91 EPM Started
05/07/2019 00:20:03.90 Booting after System Failure.
05/07/2019 00:20:02.72 Changing to watchdog warm reset mode
05/07/2019 00:12:20.23 Epm application wdg timer warning - 50 sec, kepc 0xffffffff806abae8(io_schedule+0x80/0xb0) uepc 0x76e4eccc.
05/07/2019 00:12:10.12 Epm application wdg timer warning - 40 sec, kepc 0xffffffff806abae8(io_schedule+0x80/0xb0) uepc 0x463988.
05/07/2019 00:11:59.92 Epm application wdg timer warning - 30 sec, kepc 0xffffffff806abae8(io_schedule+0x80/0xb0) uepc 0x463988.
05/07/2019 00:11:52.49 Epm application wdg timer warning - 20 sec, kepc 0xffffffff806abae8(io_schedule+0x80/0xb0) uepc 0x405190.
05/07/2019 00:11:44.21 EPM was likely not running for 29 seconds. Will take adjust actions
05/07/2019 00:11:33.54 Because the main (2006495232) thread of process 1828, has not responded within 41 periods of 1 seconds, the process will be terminated.
05/07/2019 00:10:43.24 Epm application wdg timer warning - 30 sec, kepc 0xffffffff806abae8(io_schedule+0x80/0xb0) uepc 0x463988.
05/07/2019 00:10:33.02 Epm application wdg timer warning - 20 sec, kepc 0xffffffff806abae8(io_schedule+0x80/0xb0) uepc 0x463988.
05/06/2019 23:17:46.36 CPU utilization monitor: process hal consumes 91 % CPU
05/06/2019 23:15:26.37 CPU utilization monitor: process hal consumes 91 % CPU

Thanks in advance for the help

topic Two switch X460G2 rebooted unexpectedly after a high CPU of process hal in ExtremeSwitching (EXOS/Switch Engine)

Two switch X460G2 rebooted unexpectedly after a high CPU of process hal