topic RE: Problem with memory in ExtremeSwitching (EXOS/Switch Engine)

Problem with memory

Savio — Wed, 17 Dec 2014 16:03:00 GMT

Hello, colleagues!
Everyday we have trouble with our extrereme x480( ExtremeXOS version 15.5.2.9 v1552b9-patch1-5 by release-manager ). Memory at device to decreases to zero and rebooting.

I have seen on console:
Epm application wdg timer warning - 20 sec, kepc 0xffffffff8056a4d0(__mutex_lock_slowpath+0xc8/0x1f8) uepc 0x2abdc428.
Epm application wdg timer warning - 30 sec, kepc 0xffffffff8056956c(io_schedule+0x4c/0x70) uepc 0x2abdc428.
Epm application wdg timer warning - 40 sec, kepc 0xffffffff8056956c(io_schedule+0x4c/0x70) uepc 0x2abdc428.
Epm application wdg timer warning - 50 sec, kepc 0xffffffff8056956c(io_schedule+0x4c/0x70) uepc 0x2abdc428.
Epm application wdg timer warning - 60 sec, kepc 0xffffffff8056956c(io_schedule+0x4c/0x70) uepc 0x2abdc428.
Epm application wdg timer warning - 70 sec, kepc 0xffffffff8056956c(io_schedule+0x4c/0x70) uepc 0x2abdc428.
Epm application wdg timer warning - 80 sec, kepc 0xffffffff8056956c(io_schedule+0x4c/0x70) uepc 0x2abdc428.
Epm application wdg timer warning - 90 sec, kepc 0xffffffff8056956c(io_schedule+0x4c/0x70) uepc 0x2abdc428.
Epm application wdg timer warning - 100 sec, kepc 0xffffffff8056956c(io_schedule+0x4c/0x70) uepc 0x2abdc428.
Epm application wdg timer warning - 111 sec, kepc 0xffffffff8056956c(io_schedule+0x4c/0x70) uepc 0x2abdc428.
Application watchdog killing process 1082(epm) in state 2.
Epm application wdg timer warning - 111 sec, kepc 0xffffffff8056956c(io_schedule+0x4c/0x70) uepc 0x2abdc428.
Epm application watchdog timer expired - 1203 msec. Resetting the system.
Watchdog Rebooting........

I use device as L2 and L3 (BGP,OSPF) router

x480.4 # show iproute summary
=================ROUTE SUMMARY=================
Mask distribution:
1 default routes 16 routes at length 8
12 routes at length 9 30 routes at length 10
92 routes at length 11 265 routes at length 12
504 routes at length 13 1017 routes at length 14
1764 routes at length 15 13118 routes at length 16
7264 routes at length 17 11997 routes at length 18
24990 routes at length 19 35925 routes at length 20
37837 routes at length 21 56283 routes at length 22
48837 routes at length 23 119332 routes at length 24
225 routes at length 25 153 routes at length 26
144 routes at length 27 167 routes at length 28
71 routes at length 29 170 routes at length 30
6 routes at length 31 1668 routes at length 32

Route origin distribution:
2 IBGP 361640 EBGP 230 OSPFExt2
2 Blackhole 14 Direct

Total number of routes = 361888
Total number of compressed routes = 137742

Please help me! Thanks!

RE: Problem with memory

PARTHIBAN_CHINN — Wed, 17 Dec 2014 16:33:00 GMT

show debug system-dump
show tech all
Lets see if there is any core dumps in "ls"
I would prefer moving to a better stable exos but I would recommend checking with TAC

RE: Problem with memory

Savio — Wed, 17 Dec 2014 16:35:00 GMT

x480.11 # show debug system-dump
===============================================
Switch system dump information
===============================================
core_dump_info storage: 8/3072 used [empty]
--------------------------------
System memory depleted.
time: Tue Dec 16 14:29:36 2014
Global Memory Stats:
MemTotal: 1021492 kB
MemFree: 3996 kB
Buffers: 0 kB
Cached: 6392 kB
Active(anon): 393276 kB
Inactive(anon): 393772 kB
Active(file): 3068 kB
Inactive(file): 3400 kB
HighTotal: 0 kB
HighFree: 0 kB
LowTotal 1021492 kB
LowFree: 3996 kB
Committed_AS: 945468 kB
Top Processes by RSS:
hal: 49496 kB
rtmgr: 143448 kB
dcbgp: 537248 kB
cliMaster: 15980 kB
xmld: 2740 kB
emsServer: 1912 kB
cfgmgr: 1384 kB
snmpMaster: 1404 kB
snmpSubagent: 1392 kB
fdb: 1236 kB
mcmgr: 1172 kB
ospf: 1064 kB
etmon: 1988 kB
idMgr: 1896 kB
Top Slab Caches by Memory Consumption:
size-128: 56156 kB
ip_fib_hash: 15808 kB
size-16384: 3472 kB
size-4096: 3324 kB
jffs2_refblock: 2480 kB
size-524288: 2560 kB
size-1024: 2100 kB
UNIX: 1132 kB
sock_inode_cache: 1084 kB
size-262144: 1792 kB
size-65536: 1536 kB
size-32768: 1472 kB
dentry: 932 kB
size-131072: 896 kB
vm_area_struct: 892 kB
jffs2_node_frag: 696 kB
--------------------------------
System memory depleted.
time: Tue Dec 16 14:29:41 2014
Global Memory Stats:
MemTotal: 1021492 kB
MemFree: 4324 kB
Buffers: 20 kB
Cached: 6416 kB
Active(anon): 392516 kB
Inactive(anon): 394148 kB
Active(file): 3160 kB
Inactive(file): 3156 kB
HighTotal: 0 kB
HighFree: 0 kB
LowTotal 1021492 kB
LowFree: 4324 kB
Committed_AS: 945468 kB
Top Processes by RSS:
hal: 49496 kB
rtmgr: 143448 kB
dcbgp: 537248 kB
cliMaster: 15980 kB
xmld: 2740 kB
emsServer: 1912 kB
cfgmgr: 1384 kB
snmpMaster: 1404 kB
snmpSubagent: 1392 kB
fdb: 1236 kB
mcmgr: 1172 kB
ospf: 1064 kB
etmon: 1988 kB
idMgr: 1896 kB
Top Slab Caches by Memory Consumption:
size-128: 56156 kB
ip_fib_hash: 15808 kB
size-16384: 3472 kB
size-4096: 3316 kB
jffs2_refblock: 2480 kB
size-524288: 2560 kB
size-1024: 2104 kB
UNIX: 1132 kB
sock_inode_cache: 1084 kB
size-262144: 1792 kB
size-65536: 1536 kB
size-32768: 1472 kB
dentry: 920 kB
size-131072: 896 kB
vm_area_struct: 892 kB
jffs2_node_frag: 692 kB
--------------------------------
System memory depleted.
time: Tue Dec 16 14:29:46 2014
Global Memory Stats:
MemTotal: 1021492 kB
MemFree: 3996 kB
Buffers: 4 kB
Cached: 6408 kB
Active(anon): 392516 kB
Inactive(anon): 394404 kB
Active(file): 3224 kB
Inactive(file): 3160 kB
HighTotal: 0 kB
HighFree: 0 kB
LowTotal 1021492 kB
LowFree: 3996 kB
Committed_AS: 945468 kB
Top Processes by RSS:
hal: 49500 kB
rtmgr: 143448 kB
dcbgp: 537248 kB
cliMaster: 15980 kB
xmld: 2740 kB
emsServer: 1912 kB
cfgmgr: 1384 kB
snmpMaster: 1404 kB
snmpSubagent: 1392 kB
fdb: 1236 kB
mcmgr: 1172 kB
ospf: 1064 kB
etmon: 1988 kB
idMgr: 1896 kB
Top Slab Caches by Memory Consumption:
size-128: 56156 kB
ip_fib_hash: 15808 kB
size-16384: 3472 kB
size-4096: 3316 kB
jffs2_refblock: 2480 kB
size-524288: 2560 kB
size-1024: 2092 kB
UNIX: 1132 kB
sock_inode_cache: 1084 kB
size-262144: 1792 kB
size-65536: 1536 kB
size-32768: 1472 kB
dentry: 920 kB
size-131072: 896 kB
vm_area_struct: 892 kB
jffs2_node_frag: 680 kB

x480.12 #

RE: Problem with memory

PARTHIBAN_CHINN — Wed, 17 Dec 2014 16:48:00 GMT

I dont understand why the rtmgr process is high.
Its better we reach out to TAC.
hal: 49496 kB rtmgr: 143448 kB--------high
dcbgp: 537248 kB
cliMaster: 15980 kB Either its a known issue or TAC needs to replicate to fix this issue.

RE: Problem with memory

Savio — Wed, 17 Dec 2014 17:59:00 GMT

Sorry, but what is TAC?

RE: Problem with memory

PARTHIBAN_CHINN — Wed, 17 Dec 2014 18:19:00 GMT

You should have support contract for reaching technical assistance center.
You can call in and give ur serial number.
they will help u troubleshooting this problem.
http://www.extremenetworks.com/support/contact
Check the above link

RE: Problem with memory

Savio — Sat, 20 Dec 2014 16:39:00 GMT

I found what the problem is.
after write command:

configure iproute reserved-entries maximum
all is well

RE: Problem with memory

Savio — Sat, 20 Dec 2014 16:39:00 GMT

#
# Module hal configuration.
#
configure forwarding external-tables l3-only
configure iproute reserved-entries maximum

RE: Problem with memory

Stephane_Grosj1 — Sun, 21 Dec 2014 01:56:00 GMT

That meant you were storing the routes into memory. x480 is capable to store up to 512k routes in hardware (more with compression enabled), but you have to configure the switch for that. The default is L2-and-L3, with 256k each. You were above that threshold.