topic Re: Some accesspoints (AP130 and AP230) are reporting: AP's exceed maximum number of IP sessions (98828) Is there a way to figure out where the sessions are going? in Aerohive Migrated Content

Some accesspoints (AP130 and AP230) are reporting: AP's exceed maximum number of IP sessions (98828) Is there a way to figure out where the sessions are going?

bjorn — Tue, 06 Nov 2018 21:40:25 GMT

Some accesspoints (AP130 and AP230) are reporting: AP's exceed maximum number of IP sessions (98828) Is there a way to figure out where the sessions are going?

Re: Some accesspoints (AP130 and AP230) are reporting: AP's exceed maximum number of IP sessions (98828) Is there a way to figure out where the sessions are going?

samantha_lynn — Tue, 06 Nov 2018 22:57:14 GMT

If you run "show station" that should show you the current number of connected sessions, does that seem to exceed 200? Also, are you hosting the DHCP server on an AP or on your backend network? If we are able to see, could we check your DHCP table to see if we have an unexpected number of IP addresses leased out?

Re: Some accesspoints (AP130 and AP230) are reporting: AP's exceed maximum number of IP sessions (98828) Is there a way to figure out where the sessions are going?

bjorn — Tue, 06 Nov 2018 23:09:24 GMT

show station will only show me the SSID summary

if I do "show forwarding-engine ip-sessions" i can hold my space bar for half an hour 😉

I see a lot of traffic going to port 3000 (UDP):

Id:204; Ageout:55656; Flags:0x20400; QOS:2; L7 ID:65535; ALG=n/a; Up: 0 min 44 sec;

10.x.x.122/43702 -> 10.x.x.106/3000; Proto 17; Flg:0x102; Pkts:1 Bytes:940 Parent-MAC-Sess: 15

10.x.x.106/3000 -> 10.x.x.122/43702; Proto 17; Flg:0x0; Pkts:0 Bytes:0

Re: Some accesspoints (AP130 and AP230) are reporting: AP's exceed maximum number of IP sessions (98828) Is there a way to figure out where the sessions are going?

AnonymousM — Wed, 07 Nov 2018 00:15:40 GMT

Bjorn,

What HiveOS are you running? I seem to remember a HiveOS a while back that did not terminate those sessions properly and would fill up the buffer. I believe it was only happening on the AP250s, but I can't quite remember the details. Nor can I find any of my notes on the project I was working on that had the problems. 😞

Re: Some accesspoints (AP130 and AP230) are reporting: AP's exceed maximum number of IP sessions (98828) Is there a way to figure out where the sessions are going?

bjorn — Wed, 07 Nov 2018 04:05:16 GMT

6.5r4

It is an AP130

Re: Some accesspoints (AP130 and AP230) are reporting: AP's exceed maximum number of IP sessions (98828) Is there a way to figure out where the sessions are going?

samantha_lynn — Thu, 08 Nov 2018 04:57:32 GMT

Could you tell me what your subnet size is? If we're using something above /23 or /24 that might cause that initial warning. Also, I'd like to disable updates between Hive neighbors in the same subnet to clear up some space for the IP sessions.

You can disable this (assuming you're using HiveManager, formerly NG) by going to Configure> Open the Network Policy> Go to the Additional Settings tab> Left hand side menu under Policy Settings, click on HIVE> Scroll down to the Client Roaming settings> Uncheck the box next to "Update hive members in the same subnet and VLAN". Let's leave "Update all hive members within radio range..." checked so we don't interfere with normal roaming functions.

If you could push that out to a few APs to test, and then let me know if you see any change in the IP session warnings, I would appreciate it. If you could send me tech data from an AP with this issue I can check the usual suspects to make sure nothing stands out, my email is communityhelp@aerohive.com.

Just for future reference, in case some over-zealous support agent requests you run a command with a very long output, the command "console page 0" will tell the device to display all output for any command right away, without needing to hold down the space bar forever. Sorry about that!

Re: Some accesspoints (AP130 and AP230) are reporting: AP's exceed maximum number of IP sessions (98828) Is there a way to figure out where the sessions are going?

bjorn — Thu, 08 Nov 2018 05:22:31 GMT

/24

I already have "Update hive members in the same subnet and VLAN" unchecked 😉

Tech data is send in 5 minutes

Thnx for the tip!

Re: Some accesspoints (AP130 and AP230) are reporting: AP's exceed maximum number of IP sessions (98828) Is there a way to figure out where the sessions are going?

AnonymousM — Thu, 08 Nov 2018 05:36:04 GMT

Bjorn,

The issue I remember was definitely for the AP250s and only when running one of their early code revs (7.0r2). The fix (think workaround in reality) was to run the following command (alg sip enable). This enabled a timer that would terminate unused sessions and not allow them to stay open. I can't say that you are experiencing this exact problem as your hardware and HiveOS are different, but it may be worth looking at.

If you do a show log buff or possibly show log flash, do you see any of the following output:

err kernel: [fe]: exceed maximum number of IP sessions [8191] allowed, per 80000

Sorry I can't be of much more help. I was always a big fan of 6.5r4 but we didn't deploy many AP130s. Have you tried a newer HiveOS by chance?

Re: Some accesspoints (AP130 and AP230) are reporting: AP's exceed maximum number of IP sessions (98828) Is there a way to figure out where the sessions are going?

bjorn — Thu, 08 Nov 2018 06:00:30 GMT

When I click on techdata i get a popup with processing but after a while the popup is gone and i'm back to the previous screen (popup blocker disabled and using chrome)

Am I doing something wrong?

Re: Some accesspoints (AP130 and AP230) are reporting: AP's exceed maximum number of IP sessions (98828) Is there a way to figure out where the sessions are going?

bjorn — Thu, 08 Nov 2018 06:03:08 GMT

@Brian, the customer doesn't want to upgrade because he has a stable environment right now (we had a lot of troubles with unstable firmwares in the past)

This is my syslog (from yesterday)

Nov 6 10:04:03 172.*.*.106 kernel: [fe]: exceed maximum number of IP sessions [8191] allowed, per 80000

Nov 6 13:11:29 172.*.*.106 kernel: [fe]: exceed maximum number of IP sessions [8191] allowed, per 80000

Nov 6 13:16:38 172.*.*.122 kernel: [fe]: exceed maximum number of IP sessions [8191] allowed, per 80000

Re: Some accesspoints (AP130 and AP230) are reporting: AP's exceed maximum number of IP sessions (98828) Is there a way to figure out where the sessions are going?

AnonymousM — Thu, 08 Nov 2018 06:17:05 GMT

Bjorn,

I totally get the hesitation in firmware upgrading. We lived on 6.5r4 for years and still have thousands of APs riding it for the forceable future.

The alg sip enable command may be of value but only if there are sessions that are not self terminating properly. If however there are actually that many "active" sessions, there could be another underlying issue at play.

You can manually clear them (clear forwarding-engine ip-sessions), but that doesn't really fix the problems. A band-aid for a gushing wound at best.

CPU/Memory being taxed at all to think that a loop or DoS attack or something similar could be happening by chance? Any/all of the following might shed some additional light onto things: show cpu detail | show mem | show system process state

Re: Some accesspoints (AP130 and AP230) are reporting: AP's exceed maximum number of IP sessions (98828) Is there a way to figure out where the sessions are going?

samantha_lynn — Thu, 08 Nov 2018 21:55:47 GMT

Sorry about the issues getting the tech data, that might indicate we don't have TCP 22 open on the firewall/content filters, or it could be a server issue on our end (I'll look in to that from my side), but to cut to the chase it might be faster to get the tech data via the CLI. This guide will walk you through that- https://thehivecommunity.aerohive.com/s/article/Collecting-Tech-Data-via-CLI

Re: Some accesspoints (AP130 and AP230) are reporting: AP's exceed maximum number of IP sessions (98828) Is there a way to figure out where the sessions are going?

samantha_lynn — Thu, 08 Nov 2018 23:57:19 GMT

Thanks very much for the tech data. Some notes on what I looked at/found:

CPU is not spiking

Checked for warnings related to the issue, none found

Reviewed the error messages in question, found quite a few, no further details

No evidence of a broadcast storm so far

If you could run the following commands, I'd like to check the incomming LLC count, but the AP's been up for 20 weeks so the counters would be pretty general at this point.

clear forward count int eth0

*let the AP run as normal for 2 minutes before running the next command, time it if you can because we need an accurate second count between these two commands.*

show forward count int eth0

Then if you can send me the output of that command, what we're looking for is the incoming LLC count and we'll divide that by 120 seconds to get an average packet per second count. We're hoping to see this below 30, above that we'll start seeing issues and then we'd want a packet capture to dig deeper in to where those packets are coming from.

Additionally, Brian's suggestion about running "alg sip enable" is a great one. In the event the TCP monitor function is not operating properly, this will get it back in line and might clear up those IP sessions when a TCP connection is closed, rather than leaving them there and filling up the logs.

Re: Some accesspoints (AP130 and AP230) are reporting: AP's exceed maximum number of IP sessions (98828) Is there a way to figure out where the sessions are going?

bjorn — Fri, 09 Nov 2018 05:36:58 GMT

@brian, the reason why we didn't use the alg sip enable is that our voip provider (yes we are using voice over wifi) doesn't recommend this

@Sam do i need to run those commands during the day (while there is traffic)?

Re: Some accesspoints (AP130 and AP230) are reporting: AP's exceed maximum number of IP sessions (98828) Is there a way to figure out where the sessions are going?

samantha_lynn — Fri, 09 Nov 2018 05:50:18 GMT

Yes please, we'd want to test during what would be considered normal business day traffic.

Re: Some accesspoints (AP130 and AP230) are reporting: AP's exceed maximum number of IP sessions (98828) Is there a way to figure out where the sessions are going?

bjorn — Fri, 09 Nov 2018 19:40:39 GMT

Hi Sam,

Here you go (exactly 2 minutes) 🙂

Interface [eth0]

-----------------

Inocming Counters

------------------------------

LLC: 4085; ARP: 1255; IP 8152

ICMP: 0; UDP: 7790; TCP: 119; GRE: 0; ESP: 0

HTTP: 0; HTTPS: 0; FTP: 0; TELNET: 0; DNS: 0; DHCP: 110; SSH: 4

Dropped: 9101

Outgoing Counters

------------------------------

LLC: 182; ARP: 151; IP 1263

ICMP: 0; UDP: 1134; TCP: 125; GRE: 0; ESP: 0

HTTP: 34; HTTPS: 116; FTP: 0; TELNET: 0; DNS: 13; DHCP: 0; SSH: 0

Cheers,

Bjorn

Re: Some accesspoints (AP130 and AP230) are reporting: AP's exceed maximum number of IP sessions (98828) Is there a way to figure out where the sessions are going?

samantha_lynn — Fri, 09 Nov 2018 21:56:19 GMT

Thanks very much for that output, that shows 4085 incoming LLC packets, divided by 120 seconds for the 2 minutes, that gives us an average of roughly 34 packets per second. This is over the threshold of the max we'd like to see per second, so that might be a sign of the issue causing those errors. Could we get a packet capture so I can check your traffic to see if we can find where the excess is coming from?

This guide reviews how to run a packet capture using Wireshark (alternatively, there is a built in packet capture feature now in HiveManager that doesn't need Wireshark or Cloudshark, but I'm still working on the guide for that. If you want to check it out, it's in the Tools tab)- https://thehivecommunity.aerohive.com/s/article/Packet-Capture-in-NG

Thanks very much for your patience, I'm sorry for all the different data grabs and tests, but I appreciate you working with me through all of this.

Re: Some accesspoints (AP130 and AP230) are reporting: AP's exceed maximum number of IP sessions (98828) Is there a way to figure out where the sessions are going?

bjorn — Fri, 09 Nov 2018 22:24:09 GMT

Hi Sam, no worries. I'm very much enjoying the ride 🙂

Regarding the packet capture, which interface (both wifi?), duration and Packet Count per Interface?

Cheers,

Bjorn

Re: Some accesspoints (AP130 and AP230) are reporting: AP's exceed maximum number of IP sessions (98828) Is there a way to figure out where the sessions are going?

samantha_lynn — Fri, 09 Nov 2018 23:14:06 GMT

Let's look at eth0 to start with if you could, and if we can safely assume the IP session error is occurring during the time of the packet capture, let's run it for 10 minutes and pull tech data right after we stop the packet capture so I can compare what we're seeing in both places for a wider picture.

Also, could you tell me how many VLANs your switch ports are trunked for? And if you've adjusted the roaming cache or the local cache age out times? Those would be in the SSID Additional Settings> Optional Settings> Client related network settings section.

Re: Some accesspoints (AP130 and AP230) are reporting: AP's exceed maximum number of IP sessions (98828) Is there a way to figure out where the sessions are going?

bjorn — Fri, 09 Nov 2018 23:19:21 GMT

I'm using the remote packet capture and there is no eth0 but only wifi0 and wifi1