topic Re: AP7131 possible memory leak in ExtremeWireless (WiNG)

AP7131 possible memory leak

chris_slaterwal — Fri, 22 May 2020 18:37:51 GMT

Has any noticed that AP7131 with firmware version 5.9.3.3-004R show gradually reducing free memory, to the point where it becomes impossible to log on to individual access points? We seem to be getting this problem, such that in order to restore access to an AP, it has to be rebooted.

Re: AP7131 possible memory leak

ckelly — Fri, 22 May 2020 20:13:54 GMT

Chris, how are you running a 7131 on 5.9.3? To my knowledge, the last WiNG version to support the 7131 was 5.8.6.x.

Re: AP7131 possible memory leak

chris_slaterwal — Fri, 22 May 2020 20:49:03 GMT

Well this is what I get with a “sh ver” from one of the AP7131s:

HOSTNAME*#sh ver
AP7131 version 5.9.3.3-004R
Copyright (c) 2004-2019 Extreme Networks, Inc. All rights reserved.
Booted from secondary

This shows what I mean about memory depletion:

HOSTNAME*#ser sh sys
System Information:

Free RAM: 10.9% (9924 of 90900) Min: 7.0%
File Descriptors: free: 8202 used: 649 max: 25500
CPU load averages: 1 min: 36.8% 5 min: 24.6% 15 min: 15.1%

Kernel Buffers:
Size: 32 64 128 256 512 1k 2k 4k 8k 16k 32k 64k 128k
Usage: 3780 1839 1700 1170 351 204 41 46 106 43 12 2 1
Limit: 32768 32678 4096 4096 8192 8192 16384 16384 1024 512 256 64 64

The AP was very slow to respond to every command.

Re: AP7131 possible memory leak

ckelly — Fri, 22 May 2020 21:32:10 GMT

Okay...hard to argue with that. 🙂

The only way that this could be happening then is that the controller is using the outdoor 7161 5.9.3 firmware to load onto the 7131 (7131 and 7161 are internally the same AP)...since the 7161 was in fact still supported by 5.9.3 but the 7131 was not. Whether or not the controller should have been allowed to let this happen is another topic.

Officially though, the 7131 running 5.9.3 is not supported by GTAC, but works - meaning that an issue like this would not be supported by GTAC after seeing that the 7131 is running 5.9.3 code.

Also, the 5.9.3.3 code is the last of the 5.9.3 branch, so even if there is a fix for the memory issue starting in 5.9.4, there is no firmware for the 7131/7161 once you go to 5.9.4.

I’m sure not the answer you’re looking for, but that’s the situation.

Re: AP7131 possible memory leak

Christopher_Fra — Sat, 23 May 2020 02:15:45 GMT

Though the AP7131 is no longer supported and Chris Kelly is correct regarding supported firmware, you can monitor the AP7131 via CLI to see what process is chewing up the memory:

enable [enter]

service show process [enter]

I notice that your AP7131 has crash files present (based on * at end of hostname)

You can see what crashes are present on the AP from the CLI:

enable [enter]

cd flash:/crashinfo [enter]

You can delete these files from the above directory. If many APs have crash files and are adopted to a controller, you can perform the following from the controller CLI:

enable [enter]

remote-debug clear-crash-info rf-domain xxxxx [enter] (x represents rf-domain name)

If you have multiple RF-Domains, you will need to issue the above command for each RF-Domain.

I strongly advise reaching out to your reseller and/or Extreme Networks Sales Team to possibly refresh the AP7131s with newer, supported models.

Re: AP7131 possible memory leak

chris_slaterwal — Tue, 26 May 2020 15:09:57 GMT

This is the output of “service show process” from one of the AP7131s.

I’m not sure whether the amount of memory used by cfgd is high or not?

HOSTNAME*#ser sh proc PID STATUS RSS PPID %CPU %MEM COMMAND 1387 R 29840 1 0.9 32.8 cfgd 1540 S 9088 1 1.3 9.9 dpd2 1369 S 7720 1 3.3 8.4 rim 2055 S 4032 2023 0.0 4.4 php-cgi 1430 S 3188 1 0.0 3.5 nsm 1498 S 3164 1 0.0 3.4 hsd 1338 S 2616 1 0.0 2.8 logd 2023 S 2336 1 0.0 2.5 hotspotsrvr 7961 S 2176 2003 0.0 2.3 sshd 1516 S 2172 1 0.0 2.3 mstp 2003 S 2100 1 0.0 2.3 sshd 1958 S 1964 1 0.0 2.1 ntpd 2059 S 1524 2055 0.0 1.6 php-cgi 1414 S 1380 1387 0.0 1.5 log_file 1558 S 1292 1540 0.0 1.4 log_file 7962 S 856 7961 0.0 0.9 sshd 1702 S 848 1 0.0 0.9 udhcpc 1 S 832 0 0.0 0.9 init 1983 S 656 1 0.0 0.7 telnetd 1982 S 284 1 0.0 0.3 init 2 SWN 0 1 0.0 0.0 ksoftirqd/0 3 SW 0 1 0.0 0.0 watchdog/0 4 SW< 0 1 0.0 0.0 events/0 5 SW< 0 1 0.0 0.0 khelper 8 SW< 0 6 0.0 0.0 kblockd/0 972 SW< 0 6 0.0 0.0 khubd 644 SWN 0 1 0.0 0.0 jffs2_gcd_mtd9 643 SWN 0 1 0.0 0.0 jffs2_gcd_mtd8 1109 SW< 0 6 0.0 0.0 dataplane_log 645 SWN 0 1 0.0 0.0 jffs2_gcd_mtd10 33 SW 0 6 0.0 0.0 pdflush 34 SW 0 6 0.0 0.0 pdflush 35 SW 0 1 0.0 0.0 kswapd0 36 SW< 0 6 0.0 0.0 aio/0 6 SW< 0 1 0.0 0.0 kthread 1128 SW< 0 4 0.0 0.0 ECM Thread 1129 SW< 0 4 0.0 0.0 SA_PROXY Thread 1130 SW< 0 4 0.0 0.0 FPM Thread 562 SW 0 1 0.0 0.0 mtdblockd

At least some of the AP7131s also have some files in flash:/crashinfo:

HOSTNAME*#dir flash:/crashinfo Directory of flash:/crashinfo

-rw- 1476853 Tue Apr 28 10:19:52 2020 dpdc_6_32745_1588069126_AP7131_5.9.3.3-004R.core.gz -rw- 684032 Tue Apr 28 11:08:17 2020 dpdc_6_550_1588072059_AP7131_5.9.3.3-004R.core.gz -rw- 71405 Tue Apr 28 11:07:39 2020 techsupport_low_mem_AP7131_5.9.3.3-004R.dump -rw- 51471 Tue Apr 28 11:13:03 2020 panic_202004281112_AP7131_5.9.3.3-004R.tar.gz -rw- 1472868 Tue Apr 28 10:49:06 2020 dpdc_6_335_1588070270_AP7131_5.9.3.3-004R.core.gz

Now I just need to work out a way of getting those files off the AP onto something where I can look at them...not easy with our security policy!

Thanks for your very useful replies though!

Chris.

Re: AP7131 possible memory leak

DharmaAdmin — Thu, 28 May 2020 19:01:28 GMT

Hi - The reference above cites the last supported version of WiNG on the AP7131 as 5.8.6.x

On the Exteme downloads page, I see AP 71XX Software v5.8.6.12-002R as being compatible with the AP7131.

Since these units are End-of-Life, and the firmware is hard to get, would anyone be able to post a link to download this last version (or at least somewhere in 5.8.6)? I would greatly appreciate it. I want to load it on to my AP that is the Wireless Controller and then have it distribute to a network of a dozen identical AP7131’s.

Thanks (if possible) - Frank