topic RE: X770-32q got rebooted and suddenly when i checked the monitoring i see that the memory went high. in ExtremeSwitching (EXOS/Switch Engine)

X770-32q got rebooted and suddenly when i checked the monitoring i see that the memory went high.

Dineshkumar_Ram — Fri, 09 Mar 2018 22:05:00 GMT

Suddenly my X770-32q rebooted and when i checked the monitoring i see that the memory went high and rebooted.

I have enabled the debug mode log so this make the memory to go high?

RE: X770-32q got rebooted and suddenly when i checked the monitoring i see that the memory went high.

Dineshkumar_Ram — Fri, 09 Mar 2018 22:08:00 GMT

RE: X770-32q got rebooted and suddenly when i checked the monitoring i see that the memory went high.

EtherMAN — Sat, 10 Mar 2018 22:45:00 GMT

Did you see if there was s system dump file created? This will give GTAC some clue as to what may have caused the reboot.. Command is
show debug system-dump

RE: X770-32q got rebooted and suddenly when i checked the monitoring i see that the memory went high.

Dineshkumar_Ram — Sat, 10 Mar 2018 22:45:00 GMT

Hi,

Please find the output,

X770 # show debug system-dump
===============================================
Switch system dump information
===============================================
core_dump_info storage: 8/3072 used [empty]
--------------------------------
System memory depleted.
time: Sun Mar 11 02:36:52 2018
Global Memory Stats:
MemTotal: 1021524 kB
MemFree: 4012 kB
Buffers: 16 kB
Cached: 6396 kB
Active(anon): 41416 kB
Inactive(anon): 41552 kB
Active(file): 3204 kB
Inactive(file): 3096 kB
HighTotal: 0 kB
HighFree: 0 kB
LowTotal 1021524 kB
LowFree: 4012 kB
Committed_AS: 599644 kB
Top Processes by RSS:
cliMaster: 14180 kB
hal: 9692 kB
etmon: 6264 kB
cfgmgr: 2056 kB
vlan: 2180 kB
netTools: 2500 kB
xmld: 2772 kB
idMgr: 2140 kB
epm: 1040 kB
emsServer: 1752 kB
fdb: 1656 kB
edp: 1044 kB
rtmgr: 1656 kB
mcmgr: 1632 kB
pim: 1220 kB
snmpSubagent: 1384 kB
Top Slab Caches by Memory Consumption:
size-2048: 166956 kB
size-256: 293740 kB
size-192: 204772 kB
size-128: 36700 kB
size-524288: 9216 kB
size-65536: 9792 kB
size-262144: 8192 kB
size-16384: 4304 kB
size-4096: 4524 kB
jffs2_refblock: 3864 kB
size-131072: 3200 kB
size-1024: 3900 kB
size-64: 3240 kB
UNIX: 1108 kB
sock_inode_cache: 1044 kB
size-32768: 1408 kB
--------------------------------
System memory depleted.
time: Sun Mar 11 02:36:57 2018
Global Memory Stats:
MemTotal: 1021524 kB
MemFree: 4036 kB
Buffers: 16 kB
Cached: 6344 kB
Active(anon): 41428 kB
Inactive(anon): 43148 kB
Active(file): 3168 kB
Inactive(file): 3148 kB
HighTotal: 0 kB
HighFree: 0 kB
LowTotal 1021524 kB
LowFree: 4036 kB
Committed_AS: 599644 kB
Top Processes by RSS:
cliMaster: 14196 kB
hal: 9700 kB
etmon: 6264 kB
cfgmgr: 2056 kB
vlan: 2180 kB
netTools: 2500 kB
xmld: 2776 kB
idMgr: 2148 kB
epm: 1040 kB
emsServer: 1752 kB
fdb: 1656 kB
edp: 1044 kB
rtmgr: 1664 kB
mcmgr: 1632 kB
pim: 1228 kB
snmpSubagent: 1384 kB
Top Slab Caches by Memory Consumption:
size-2048: 166956 kB
size-256: 293740 kB
size-192: 204772 kB
size-128: 36700 kB
size-524288: 9216 kB
size-65536: 9792 kB
size-262144: 8192 kB
size-16384: 4304 kB
size-4096: 4336 kB
jffs2_refblock: 3864 kB
size-131072: 3200 kB
size-1024: 3900 kB
size-64: 3240 kB
UNIX: 1108 kB
sock_inode_cache: 1044 kB
size-32768: 1408 kB
--------------------------------
System memory depleted.
time: Sun Mar 11 02:37:02 2018
Global Memory Stats:
MemTotal: 1021524 kB
MemFree: 4016 kB
Buffers: 16 kB
Cached: 6420 kB
Active(anon): 37128 kB
Inactive(anon): 47272 kB
Active(file): 3272 kB
Inactive(file): 3084 kB
HighTotal: 0 kB
HighFree: 0 kB
LowTotal 1021524 kB
LowFree: 4016 kB
Committed_AS: 599644 kB
Top Processes by RSS:
cliMaster: 14180 kB
hal: 9692 kB
etmon: 6264 kB
cfgmgr: 2056 kB
vlan: 2180 kB
netTools: 2500 kB
xmld: 2772 kB
idMgr: 2140 kB
epm: 1040 kB
emsServer: 1752 kB
fdb: 1656 kB
edp: 1044 kB
rtmgr: 1656 kB
mcmgr: 1632 kB
pim: 1220 kB
snmpSubagent: 1384 kB
Top Slab Caches by Memory Consumption:
size-2048: 166960 kB
size-256: 293740 kB
size-192: 204772 kB
size-128: 36704 kB
size-524288: 9216 kB
size-65536: 9792 kB
size-262144: 8192 kB
size-16384: 4304 kB
jffs2_refblock: 3864 kB
size-131072: 3200 kB
size-4096: 3872 kB
size-1024: 3968 kB
size-64: 3240 kB
UNIX: 1108 kB
sock_inode_cache: 1044 kB
size-32768: 1408 kB

RE: X770-32q got rebooted and suddenly when i checked the monitoring i see that the memory went high.

Bin — Mon, 12 Mar 2018 12:57:00 GMT

Hi, what is EXOS firmware version?

Could you also observe the state of "show cpu-monitoring" ?
We have one software issue on EXOS.

Please see the following article.
https://gtacknowledge.extremenetworks.com/articles/Solution/High-HAL-CPU-on-X770-with-EXOS-22-2-1

Best regards,

RE: X770-32q got rebooted and suddenly when i checked the monitoring i see that the memory went high.

Dineshkumar_Ram — Mon, 12 Mar 2018 12:57:00 GMT

X770# show cpu-monitoring

CPU Utilization Statistics - Monitored every 5 seconds
-----------------------------------------------------------------------

Process 5 10 30 1 5 30 1 Max Total
secs secs secs min mins mins hour User/System
util util util util util util util util CPU Usage
(%) (%) (%) (%) (%) (%) (%) (%) (secs)
-----------------------------------------------------------------------

System 13.9 15.4 15.6 45.8 22.5 17.1 19.9 85.7 95.99 58725.83
aaa 0.0 0.0 0.0 0.1 0.0 0.0 0.0 0.5 3.90 3.97
acl 0.0 0.0 0.0 0.1 0.1 0.0 0.0 1.2 38.90 97.89
bfd 0.1 0.0 0.0 0.1 0.1 0.0 0.0 0.8 38.87 56.57
bgp 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00
brm 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.8 1.19 0.89
ces 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.8 1.26 0.79
cfgmgr 0.1 0.8 0.2 2.2 0.9 0.8 0.8 20.3 956.80 441.11
cli 0.0 0.0 0.0 0.6 0.1 0.0 0.0 29.2 7.11 3.65
devmgr 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.4 5.72 5.75
dirser 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.2 0.83 0.93
dosprotect 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.5 0.11 0.09
dot1ag 0.2 0.9 0.3 0.0 0.2 0.2 0.2 2.9 60.67 14.69
eaps 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.9 1.41 1.61
edp 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.4 28.38 11.81
elrp 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.1 1.29 0.97
elsm 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.8 1.97 2.54
ems 0.0 0.0 0.0 0.0 0.0 0.0 0.0 9.6 1.86 1.42
epm 0.1 0.0 0.0 0.1 0.1 0.1 0.1 14.2 158.61 54.95
erps 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.6 2.51 1.69
esrp 0.0 0.0 0.0 0.1 0.0 0.0 0.0 1.2 7.84 13.68
ethoam 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.9 1.49 1.35
etmon 0.1 0.0 0.0 0.1 0.1 0.1 0.1 1.6 39.68 33.99
exacl 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00
exdhcpsnoop 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00
exdos 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00
exfib 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00
exfipSnoop 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00
exosmc 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00
exosq 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00
exsflow 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00
exsnoop 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00
exsshd 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.6 0.35 0.67
exvlan 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00
fcoe 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.5 2.07 1.10
fdb 0.5 0.5 0.4 0.2 0.3 0.3 0.3 6.1 953.44 351.70
gptp 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00
hal 5.7 5.0 3.4 3.7 3.8 3.7 3.8 64.2 1514.10 9619.55
hclag 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.7 1.07 0.95
idMgr 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.4 1.23 0.89
ipSecurity 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 1.09 1.26
ipfix 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.7 1.26 0.80
isis 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00
lacp 0.0 0.0 0.0 0.0 0.1 0.0 0.0 1.2 3.83 4.13
liftd 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.5 0.13 0.14
lldp 0.0 0.0 0.0 0.6 0.2 0.0 0.0 2.6 27.43 10.67
mcmgr 0.0 0.0 0.0 1.7 0.6 0.2 0.2 7.9 95.92 192.78
mpls 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00
mrp 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.1 2.26 1.72
msdp 0.0 0.0 0.0 0.0 0.1 0.0 0.0 1.0 16.53 16.13
netLogin 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 2.45 1.36
netTools 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.1 2.64 1.50
nettx 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00
nodemgr 0.0 0.0 0.0 0.1 0.1 0.1 0.0 0.2 32.35 51.11
ntp 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.6 1.38 1.46
ospf 0.0 0.0 0.0 1.7 0.4 0.2 0.2 2.3 65.31 32.14
ospfv3 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.2 7.94 7.82
pim 0.0 0.0 0.0 0.2 0.1 0.1 0.0 1.4 89.13 18.30
poe 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.7 1.06 0.78
polMgr 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.6 0.23 0.24
ptpV2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00
pwmib 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.4 0.14 0.14
rip 0.0 0.0 0.0 0.1 0.1 0.0 0.0 0.9 19.49 36.61
r.png 0.0 0.0 0.0 0.1 0.0 0.1 0.1 0.8 16.95 20.38
rtmgr 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.6 7.30 4.22
snmpMaster 0.6 0.3 1.4 6.7 3.8 3.3 3.4 19.9 5070.68 1062.51
snmpSubagent 0.5 0.2 1.3 9.4 4.6 4.0 4.1 24.9 6896.65 894.53
stp 0.0 0.0 0.0 1.2 0.2 0.2 0.1 3.2 205.16 77.19
synce 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 1.71 2.07
techSupport 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.5 0.88 0.85
telnetd 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.5 0.10 0.09
tftpd 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.4 0.11 0.15
thttpd 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.7 0.12 0.09
trill 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00
upm 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.6 0.77 0.82
vlan 0.0 0.0 0.0 1.7 0.9 0.9 1.0 8.3 1531.14 345.07
vmt 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.7 1.33 1.16
vrrp 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.7 1.56 0.84
vsm 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 1.41 1.17
xmlc 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.5 0.13 0.09
xmld 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.9 0.55 0.29

X770# show version
Switch : 800534-00-04 1442N-41167 Rev 4.0 BootROM: 2.0.2.1 IMG: 15.6.3.1
PSU-1 : Internal PSU-1 800462-00-06 1452W-80418
PSU-2 : Internal PSU-2 800462-00-06 1452W-80467

Image : ExtremeXOS version 15.6.3.1 v1563b1-patch1-5 by release-manager
on Sun Aug 16 21:34:14 EDT 2015
BootROM : 2.0.2.1
Diagnostics : 6.3

RE: X770-32q got rebooted and suddenly when i checked the monitoring i see that the memory went high.

Dineshkumar_Ram — Mon, 12 Mar 2018 12:57:00 GMT

In monitoring when i see the CPU looks good.