topic Re: Evaluating High Collision Issues in Aerohive Migrated Content

Evaluating High Collision Issues

tandrews1 — Thu, 24 Jan 2019 01:37:35 GMT

I've looked through most of the available documentation I can find here including the Radio Frequency Interference Troubleshooting Guide (https://thehivecommunity.aerohive.com/s/article/Radio-Frequency-Interference-Troubleshooting?t=1545053626774). But am still perplexed by what I am seeing in some of our environments. I have policies set up that use the recommendations in the aforementioned guide, but I still see high collision issues across the enterprise.

I'm looking at a 250 in an elementary building right now. Summary State=High Collision. This AP was alerting earlier today and the teacher reported poor performance with a cart of Chromebooks. As this is a new building I had not made any adjustments yet, but in this area I applied radio policies to three APs in close proximity that have rectified similar issues in other buildings.

I also noted that one AP was definitely a bit too loud on 2.4 so I turned down the power. Now, looking at acsp neighbors the loudest I see is -67 on channel 6 on an AP two rooms away with that radio cranked down to 4. Nothing on 5ghz worse than -75.

Band steering is enabled, but all of these devices associate on 5ghz naturally. I see very little 2.4 activity.

So right now I have two clients associated. Both on wifi1.1. I cleared counters on wifi0 and wifi1 and let it go for 3 minutes. Then I show wifi0 _count and see:

0% rx retry rate

10833 rx CRC errors

85% rx CRC rate

And show int wifi1 _count:

3% rx retry rate

104 rx CRC errors

14% rx CRC rate

And now show int wifi0 and wifi1 show a summary state of "High Collision".

I'm baffled.

Re: Evaluating High Collision Issues

samantha_lynn — Thu, 24 Jan 2019 01:47:44 GMT

The high CRC rate could be environmental, are there any sources of metal, glass, or water that the signal would need to pass through to reach the client devices?

Overall you are seeing that summary state because the CRC rate is over 10%. There are five possible states for the WiFI interfaces:

Good: The CRC and Tx error rates are less than 5%, and channel utilization is less than 50%.

Fair : The CRC and Tx error rates are less than 10%, and channel utilization is less than 65%.

Channel RF Overcrowded: The channel utilization rate is 65% or greater.

High Collision: The CRC error rate is 10% or greater.

High Tx Error: The transmit error ration is more than 10%, which can occur when the RF environment is overcrowded and the collision rate is high.

Also you mentioned you have carts of Chromebooks in the network, have you read the guide regarding Chromebooks? They have a feature called peer to peer updating that can cause some significant broadcast increases, which might explain some of the packet damage you are seeing. The guide I'm referring to can be found here- https://thehivecommunity.aerohive.com/s/article/Chromebooks-and-WiFi

Re: Evaluating High Collision Issues

sderikonja1 — Thu, 24 Jan 2019 03:06:28 GMT

Could you post a picture of the AP and surrounding?

Re: Evaluating High Collision Issues

tandrews1 — Thu, 24 Jan 2019 03:45:13 GMT

I have 265 access points spread across 15 buildings. There are definitely some RF challenges in some of them, but the building in this example is new construction. APs are all hanging from ceiling grid. Walls are mostly sheet rock on metal studs, but there are some block walls scattered around.

If environment is the issue, then I am probably going to have to live with this since I am hard-pressed to find an AP right now in this enterprise that isn't showing high collision on both interfaces.

I have read through the Chromebook suggestions and we disabled interstation traffic a while ago. We also disabled automatic updates or in some cases scattered update times for our chromebooks. We are avoiding DFS channels for now.

Re: Evaluating High Collision Issues

samantha_lynn — Thu, 24 Jan 2019 04:43:01 GMT

We do offer site surveys, which would help you adjust the network to the environment as much as possible. That is a paid service and you'd want to talk to our sales team if you're interested. If you'd like to send some pictures of a few typically placed APs, with as much of the surrounding environment shown as possible, to communityhelp@aerohive.com then I could let you know if I see any major problems and how we could work around them?

Re: Evaluating High Collision Issues

w1f1n00b — Tue, 01 Oct 2019 20:57:05 GMT

I'm curious if you ever got to the bottom of this, as I'm also dealing with similar issues. Here are some of the numbers I'm seeing. - https://thehivecommunity.aerohive.com/s/question/0D50c00007R9NzSCAV/ap550-high-rx-crc-rate-42-on-only-one-radio

Re: Evaluating High Collision Issues

jose_gonzalez — Thu, 13 Feb 2020 04:09:36 GMT

I have been looking into this in a school district. During the day & night, checking high schools with over 100 APs in a one-to-one environment, I see the high collision rates on the 2.4ng wifi0 interface. I have recorded once over 99% of the radios at specific campus / point in time with wifi0 at 'high collision'. As Sam Pirok stated due to >10% CRC RX errors. Sound the alarm bells, NOT!

I did some testing, I logged at night when no one is at the campuses, with no clients connected to the APs, cleared the counters, still high RX CRC. No problem, it's probably radio interference, 3 channels, 1 to 1 environment - so I turned all down all the 2.4 radios power to level 5, to my surprise no difference. I've tried other things, such as increase the minimum SSID data rate to 18Mbps, enable high density radio settings, deny 802.11B clients, suppress beacon responses to low SNR clients. Nothing made a difference. Now here's the kicker, take that same campus with 90%+ 2.4Ghz radios in high collision rates, with 1000+ clients connected campus wide and look at the per client state using 'show station' - and behold 11.53% clients show high collision, 8.09% fair, 80.38% good overall.

Point I'm making is not fret over this statistic, use 'show station' to see individual client performance, as I noted the CRC RX error counters runs even without any clients connected, building unoccupied. If someone complains of poor performance don't forget your basics, any slowness by a end user will be reported as wireless. If they are running a Windows PC - the slowness is more likely be operating system related, hard disk usage and or CPU usage at 100%. Don't just take the end users account as firsthand, investigate. On the wireless - if you are running AP230 / AP330 disable QoS policies to save CPU power, using CLI supplicant 'no qos enable no-prompt'. On the Radio profiles disable background scanning if any clients are connected.

Re: Evaluating High Collision Issues

sderikonja1 — Fri, 14 Feb 2020 04:10:38 GMT

Jose,

What HiveOS version and what management platform? Do you ever get reports that clients cannot associate until an AP is restarted?

Re: Evaluating High Collision Issues

jose_gonzalez — Fri, 14 Feb 2020 04:56:51 GMT

6.5r11 on AP220 and 6.5r12 on AP330. We are running on on-premise HM Classic latest revision. The max code version we can go up to is 8.2 code. We tested 8.2 code and determined that it is 'heavier' (higher CPU usage) than 6.5 code. 6.5 code outperforms 8.2 codes in high density environments. Our Aerohive representatives tell us that 10.x code runs more efficiently, but currently we can not test it - as we need HM NG.

As far as APs having to be rebooted to get clients to authenticate, no that does not occur in our environment. Many years ago, with older 6.5 code there was DHCP issues that prevented clients from authenticating until the AP was rebooted. I also remember weirdness with clients not being able to get to the gateway with Proxy ARP enabled. Rebooting the AP worked around that, however the solution was disabling proxy ARP. I recommend disabling proxy ARP, if enabled.

Re: Evaluating High Collision Issues

w1f1n00b — Fri, 14 Feb 2020 05:05:03 GMT

I've seen reboot resolve strange issues on several occasions typically with ap230's but this could be anecdotal as many of my sites have a majority of that model. I've never been able to get to the bottom of it as I'm typically trying to get things working as a higher priority to solving the underlying issue unfortunately. I've seen strangeness like this all the way up to 10.0r7a.

Re: Evaluating High Collision Issues

sderikonja1 — Mon, 17 Feb 2020 05:56:34 GMT

Jose,

I have played with Proxy ARP and it did not help resolve the association issue.

When it comes to CPU efficiency I could not tell a difference between 6.x, 8.x or 10.x. What I did observe is a high number of DFS events on 6.x, while 8.4 and 10.x would cause APs to stop moving any traffic on the wired interface. Even the serial console port is rendered unavailable. This applies to both AP230s and AP250s

John,

Not sure if the issue is the same but resulting action certainly is. The priority is after all to ensure clients can connect and take care of daily business. It bothered me though as I could not fix the problem, to at least know about it before the users report the issue, as sometimes it would take several days before they submit a ticket. I setup a Graylog server and started analyzing syslog messages to see if the misbehaving AP can be discovered sooner than later. I have come across a message like "kernel: [wifi]: wl1: 201 rx fifo 0 overflows!". Whenever an interface is generating hundreds of these messages per hour, the AP tends to refuse client association. I have a ticket open with support while I am waiting for another AP to start acting up. Problem is that most of the time I have to reboot the AP and skip troubleshooting for the sake of availability.

Re: Evaluating High Collision Issues

sderikonja1 — Fri, 06 Mar 2020 04:56:11 GMT

Jose,

For issues with Proxy-ARP, what kind of troubleshooting and analysis was done to determine that it affected ARP functionality?

Re: Evaluating High Collision Issues

jose_gonzalez — Fri, 06 Mar 2020 05:24:32 GMT

It was a few year ago, so I am vague on details. It involved the clients showing the wrong MAC address of the default gateway. Verification was done using Wireshark packet capture. It was easy to spot because the clients never received a ARP replies and in the packet details it clearly had the wrong MAC address of the router listed. Proxy ARP is never needed on properly routed network. It is legacy and was originally intended for misconfigured networks.

Re: Evaluating High Collision Issues

sderikonja1 — Sat, 07 Mar 2020 02:07:15 GMT

Jose,

Thanks for the info.