Header Only - DO NOT REMOVE - Extreme Networks
Question

AP7131 possible memory leak


Has any noticed that AP7131 with firmware version 5.9.3.3-004R show gradually reducing free memory, to the point where it becomes impossible to log on to individual access points? We seem to be getting this problem, such that in  order to restore access to an AP, it has to be rebooted.


6 replies

Userlevel 6

Chris, how are you running a 7131 on 5.9.3?  To my knowledge, the last WiNG version to support the 7131 was 5.8.6.x.

Well this is what I get with a “sh ver” from one of the AP7131s:

 

HOSTNAME*#sh ver
AP7131 version 5.9.3.3-004R
Copyright (c) 2004-2019 Extreme Networks, Inc. All rights reserved.
Booted from secondary

 

This shows what I mean about memory depletion:

HOSTNAME*#ser sh sys
System Information:

Free RAM:  10.9% (9924 of 90900)  Min: 7.0%
File Descriptors: free: 8202 used: 649 max: 25500
CPU load averages: 1 min: 36.8%  5 min: 24.6%  15 min: 15.1%

Kernel Buffers:
Size:      32    64   128   256   512    1k    2k    4k    8k   16k   32k   64k  128k
Usage:   3780  1839  1700  1170   351   204    41    46   106    43    12     2     1
Limit:  32768 32678  4096  4096  8192  8192 16384 16384  1024   512   256    64    64

 

The AP was very slow to respond to every command.

Userlevel 6

Okay...hard to argue with that. :)

The only way that this could be happening then is that the controller is using the outdoor 7161 5.9.3 firmware to load onto the 7131 (7131 and 7161 are internally the same AP)...since the 7161 was in fact still supported by 5.9.3 but the 7131 was not.  Whether or not the controller should have been allowed to let this happen is another topic.

Officially though, the 7131 running 5.9.3 is not supported by GTAC, but works - meaning that an issue like this would not be supported by GTAC after seeing that the 7131 is running 5.9.3 code.

Also, the 5.9.3.3 code is the last of the 5.9.3 branch, so even if there is a fix for the memory issue starting in 5.9.4, there is no firmware for the 7131/7161 once you go to 5.9.4.

 

I’m sure not the answer you’re looking for, but that’s the situation.

 

Userlevel 5

Though the AP7131 is no longer supported and Chris Kelly is correct regarding supported firmware, you can monitor the AP7131 via CLI to see what process is chewing up the memory:

enable [enter]

service show process [enter]

I notice that your AP7131 has crash files present (based on * at end of hostname)

You can see what crashes are present on the AP from the CLI:

enable [enter]

cd flash:/crashinfo [enter]

You can delete these files from the above directory. If many APs have crash files and are adopted to a controller, you can perform the following from the controller CLI:

enable [enter]

remote-debug clear-crash-info rf-domain xxxxx [enter] (x represents rf-domain name)

If you have multiple RF-Domains, you will need to issue the above command for each RF-Domain. 

I strongly advise reaching out to your reseller and/or Extreme Networks Sales Team to possibly refresh the AP7131s with newer, supported models. 

This is the output of “service show process” from one of the AP7131s.

I’m not sure whether the amount of memory used by cfgd is high or not?

HOSTNAME*#ser sh proc

   PID  STATUS    RSS   PPID   %CPU   %MEM   COMMAND

  1387     R    29840      1    0.9   32.8   cfgd

  1540     S     9088      1    1.3    9.9   dpd2

  1369     S     7720      1    3.3    8.4   rim

  2055     S     4032   2023    0.0    4.4   php-cgi

  1430     S     3188      1    0.0    3.5   nsm

  1498     S     3164      1    0.0    3.4   hsd

  1338     S     2616      1    0.0    2.8   logd

  2023     S     2336      1    0.0    2.5   hotspotsrvr

  7961     S     2176   2003    0.0    2.3   sshd

  1516     S     2172      1    0.0    2.3   mstp

  2003     S     2100      1    0.0    2.3   sshd

  1958     S     1964      1    0.0    2.1   ntpd

  2059     S     1524   2055    0.0    1.6   php-cgi

  1414     S     1380   1387    0.0    1.5   log_file

  1558     S     1292   1540    0.0    1.4   log_file

  7962     S      856   7961    0.0    0.9   sshd

  1702     S      848      1    0.0    0.9   udhcpc

     1     S      832      0    0.0    0.9   init

  1983     S      656      1    0.0    0.7   telnetd

  1982     S      284      1    0.0    0.3   init

     2     SWN      0      1    0.0    0.0   ksoftirqd/0

     3     SW       0      1    0.0    0.0   watchdog/0

     4     SW<      0      1    0.0    0.0   events/0

     5     SW<      0      1    0.0    0.0   khelper

     8     SW<      0      6    0.0    0.0   kblockd/0

   972     SW<      0      6    0.0    0.0   khubd

   644     SWN      0      1    0.0    0.0   jffs2_gcd_mtd9

   643     SWN      0      1    0.0    0.0   jffs2_gcd_mtd8

  1109     SW<      0      6    0.0    0.0   dataplane_log

   645     SWN      0      1    0.0    0.0   jffs2_gcd_mtd10

    33     SW       0      6    0.0    0.0   pdflush

    34     SW       0      6    0.0    0.0   pdflush

    35     SW       0      1    0.0    0.0   kswapd0

    36     SW<      0      6    0.0    0.0   aio/0

     6     SW<      0      1    0.0    0.0   kthread

  1128     SW<      0      4    0.0    0.0   ECM Thread

  1129     SW<      0      4    0.0    0.0   SA_PROXY Thread

  1130     SW<      0      4    0.0    0.0   FPM Thread

   562     SW       0      1    0.0    0.0   mtdblockd

 

At least some of the AP7131s also have some files in flash:/crashinfo:

 

 

HOSTNAME*#dir flash:/crashinfo

Directory of flash:/crashinfo

  -rw-   1476853   Tue Apr 28 10:19:52 2020   dpdc_6_32745_1588069126_AP7131_5.9.3.3-004R.core.gz

  -rw-   684032    Tue Apr 28 11:08:17 2020   dpdc_6_550_1588072059_AP7131_5.9.3.3-004R.core.gz

  -rw-   71405     Tue Apr 28 11:07:39 2020   techsupport_low_mem_AP7131_5.9.3.3-004R.dump

  -rw-   51471     Tue Apr 28 11:13:03 2020   panic_202004281112_AP7131_5.9.3.3-004R.tar.gz

  -rw-   1472868   Tue Apr 28 10:49:06 2020   dpdc_6_335_1588070270_AP7131_5.9.3.3-004R.core.gz

Now I just need to work out a way of getting those files off the AP onto something where I can look at them...not easy with our security policy!

 

Thanks for your very useful replies though!

 

Chris.

 

Hi - The reference above cites the last supported version of WiNG on the AP7131 as 5.8.6.x

On the Exteme downloads page, I see AP 71XX Software v5.8.6.12-002R as being compatible with the AP7131. 

Since these units are End-of-Life, and the firmware is hard to get, would anyone be able to post a link to download this last version (or at least somewhere in 5.8.6)? I would greatly appreciate it. I want to load it on to my AP that is the Wireless Controller and then have it distribute to a network of a dozen identical AP7131’s.

Thanks (if possible) - Frank

Reply