topic Radio Profile - channel selection auto - neighbor APs on same channel & running at Max power allowed. in ExtremeCloud IQ

Radio Profile - channel selection auto - neighbor APs on same channel & running at Max power allowed.

w1f1n00b — Wed, 18 Nov 2020 01:29:04 GMT

I’ve been seeing many instances where APs are on the same channel as their neighboring AP with RSSI in the 60s or 70s. I’m looking for steps to take to prevent this that don’t require manual channel plans, as I have too many APs to manage in that manner (wish this wasn’t the case).

In this one example I’m trying to figure out why the two radios are showing drastically different channel costs. full disclosure, not really sure what these numbers mean or how they’re calculated so any info or resources are greatly appreciated.

Also of note, these Radio Profiles are set to auto power selection with max power of 14dB. There are 50+ APs at the site (apx every other room), yet every AP is running at power 14. none have adjusted power down at all. Is this typical? My understanding is that even if Co-channel and adjacent channel interference are minimized, RF should still be kept to minimum power necessary to avoid general interference.

Re: Radio Profile - channel selection auto - neighbor APs on same channel & running at Max power allowed.

SamPirok — Wed, 18 Nov 2020 02:31:40 GMT

Hi John, starting with channel costs, the lower the channel cost the better. Channel cost goes up with more usage/traffic/noise, so a lower channel cost indicates a clearer channel with less interference, in general. This channel cost is what the auto channel selection feature uses to select the best channel for your APs to be on. Something to keep in mind with output is that it can only take in to account Aerohive APs, so if you are sharing air space with a neighboring network or individual hotspots, this output may not be totally accurate. Another thing to keep in mind is that this information is updated when the AP performs a successful background scan, and I think that’s probably where your issues is coming up.

If the AP is set to automatically choose the channel and power, it needs to be able to get a background scan through to make a decent judgement of what settings are needed in the moment. The background scan also updates the output you’re looking at now. If the AP isn’t getting background scans through, it’s going to make the decision based on the last successful scan. If you go in to your Radio Profile, can you tell me what settings you’re using for background scans?

Re: Radio Profile - channel selection auto - neighbor APs on same channel & running at Max power allowed.

w1f1n00b — Thu, 19 Nov 2020 02:17:07 GMT

Re: Radio Profile - channel selection auto - neighbor APs on same channel & running at Max power allowed.

SamPirok — Thu, 19 Nov 2020 06:07:39 GMT

Thanks for that output, those are the background settings I’d recommend so we’re good there. We got some of our senior engineers to review the output you’ve shared and it doesn’t look like the APs would need to lower their power at the moment. We usually don’t see issues until we hit the -50’s RSSI range and up, so those -66 RSSI values on channel 48 shouldn’t be causing any issues. From the outputs you’ve shared here, it looks like the radios are balancing properly at the moment.

As for the disparate channel cost values, if the AP is acting in dual 5GHz mode it’ll split the 5GHz channels between the two radios, so it’s likely that the higher channel costs are associated with the channels that radio is leaving for the other radio to use instead.

Re: Radio Profile - channel selection auto - neighbor APs on same channel & running at Max power allowed.

w1f1n00b — Mon, 23 Nov 2020 22:45:54 GMT

If Co-channel interference is “the top cause of needless airtime consumption “ (https://www.extremenetworks.com/extreme-networks-blog/whats-the-biggest-cause-of-co-channel-interference/ ) and it’s often recommended to have 20dB RSSI channel separation (https://www.metageek.com/training/resources/adjacent-channel-congestion.html), that leaves this area as an issue for any client not connected to ch48 at better than -46dB RSSI. I haven’t surveyed this particular location, but I think it’s save to say this separation would become even less likely for clients located in between these two APs on ch 48.

Given all that and my current understanding, I’m trying to wrap my head around how having two radios on the same channel so close to each other isn’t a problem?

Also of note, channels 44 & 153 aren’t even being used by any APs in the area. Any ideas why this would be?

Re: Radio Profile - channel selection auto - neighbor APs on same channel & running at Max power allowed.

jose_gonzalez — Tue, 24 Nov 2020 03:04:45 GMT

I recommend use the DFS channels so ACSP has more channels to choose from. Make sure ACSP is able to scan radio channels regardless of number of connected clients during a time range, like 12:00a to 4:00a, in the radio profile settings. Set radio transmission power floor to 2, power max drop to 18. You haven’t said what model AP and HiveOS version? 8.2r6 / 10.x is better at channel selection than 6.5, however the former use more CPU, can be a issue with large LANs and APs (AP2x0/3x0) management is on the same network. In addition I recommend you enable MU-MIMO to improve performance and disable allow bgscans with clients connected. Clients have to be disconnected in order for bgscan to run. Safest is bgscan only with no clients connected.

Regards,

Re: Radio Profile - channel selection auto - neighbor APs on same channel & running at Max power allowed.

w1f1n00b — Tue, 24 Nov 2020 03:41:02 GMT

I’d love to, but I stopped using DFS channels recently after being told by GTAC that radios will not perform background scans while on a DFS channel, effectively making it blind to ACSP (even worse if using “use last known channel/power as they’ll never evaluate the channel they’re on or the neighborhood). They’re set to adjust between 12a-6a so plenty of time there.

Examples above are AP550. I also have a lot of AP250 & AP230 all running 10.x. I have less of the 120,330,350,390 running 6.5r12

Curious what your experience is with MU-MIMO, as this is the first time I’ve seen it recommended. Its off by default, and my understanding was it was only ideal in large venues like stadiums, gyms, auditoriums etc.

I had “skip background scans when clients are connected” checked for a while as I suspected it may be causing client issues, however I’ve since unchecked that at the recommendation of GTAC. main problem I have is with clients that are online/connected 24/7 due to being in charging carts etc. Curious what your experience is with this was well.

@jose.gonzalez I appreciate your feedback, and am always looking for was to improve my environment.

Re: Radio Profile - channel selection auto - neighbor APs on same channel & running at Max power allowed.

jose_gonzalez — Tue, 24 Nov 2020 05:06:04 GMT

The radio1 interface has to perform radar background scans to use DFS channels, that’s the whole thing about DFS. ACSP does scan and evaluate the DFS channels if enabled when in channel selection mode. GTAC gets it wrong from time to time. A/C is all about MU-MIMO, if you want your AC / AX clients to experience the performance they were designed for, enable it on radio1.

If your in a dense environment I would consider allowing the channel switch at a particular time range, I charge carts too and it’s no big deal for me, the clients would connect to a neighboring AP. APs will choose a random time within the time range to channel scan. If you really can’t then make the window scan range as close to 24 hours as possible when 0 clients are connected. This AP is 20mhz, make sure all APs in dense areas are at the bandwidth.

I also can’t believe that ACSP would blindly select a channel on startup without performing channel scan first. It should perform a normal channel then based on lowest channel cost select a channel, selecting the last known channel in cases of ties.

Did you know you can force APs into ACSP channel selection mode in the CLI?

ap# show int wifi1 | inc mhz
Freq(Chan)=5180Mhz(36); EIRP power=25.77dBm; Diversity=disabled;
Channel width=20Mhz; HT-protection=Green field; Deny client=none;

ap# int wifi1 radio channel 36
ap# int wifi1 radio channel auto

Re: Radio Profile - channel selection auto - neighbor APs on same channel & running at Max power allowed.

jose_gonzalez — Tue, 24 Nov 2020 08:18:03 GMT

I noticed in your screenshots that both radios are in 5ghz mode. I discovered some type of bug in the AP410c 10.2r2 code that prevents the SDR profiles from working correctly and results in radio0 always choosing 5g. If you are running 10.2.2 go back to 10.1.6, or don’t use SDR profiles. I opened a ticket about a month ago, with all the GTAC data they needed to validate wrapped, bow tied, they are still working on validation. They don’t have the h/w to test with.

You can mitigate CVE-2020-16152 with 10.1.6 with CLI supplicant:

no system web-server hive-UI enable

Re: Radio Profile - channel selection auto - neighbor APs on same channel & running at Max power allowed.

w1f1n00b — Tue, 24 Nov 2020 22:50:12 GMT

I’ve never used SDR. I’m running a device template configured for dual 5GHz at sites that no longer need 2.4GHz support where AP250 & AP550s have been installed.

I’m still skeptical about MU-MIMO. I’ve heard time and time again that it was mostly a marketing gimmick as far as how good it functions in the real world. Maybe it’s an Aerohive specific issue/firmware problem. I don’t see how such an important feature would be disabled by default and recommend against (I’ve heard it directly from the mouth of David Coleman). But again I’m open to the possibility. This is something I may need to do further testing on.

Regarding clients being connected, I really started playing with this after noticing hundreds of channel change events at most sites throughout the coarse of the day (I’ve since reverted my templates to allow bg scans while clients are connected). I played around with different settings and determined that even with DCS (dynamic channel selection) turned off, APs were still changing channel. In the one instance where I worked with GTAC their recommendation was to manually adjust radio powers to reduce interference. We tested in a building with an AP in every room. Radio profile was set to max power 16 (dual 5ghz). All APs were running at power 16. We worked for a few weeks making different changes and adjustments, and ended up with power set manually to 2dB (down from 16, thats a huge difference). At that point the channel changes become mostly non-existent. However I don’t have the time to support manual configs on thousands of APs, and currently rely heavily on auto radio settings which is why I’m here trying to determine what is acceptable.

I feel like I’m between a rock and a hard place. In some cases I’m being told the auto channel power settings are fine, and in others I’m being told they’re the cause of channel changes outside of maintenance windows due to high interference (sometimes dozens per AP per day). All this on top of often finding APs on the same channels as neighbors, almost never seeing 5GHz radios pull their power down, and ongoing false high-interference alarms really just leaves me feeling skeptical.

Re: Radio Profile - channel selection auto - neighbor APs on same channel & running at Max power allowed.

jose_gonzalez — Wed, 25 Nov 2020 00:18:16 GMT

Don’t allow APs to switch channels if thresholds exceed x percentage in the radio profile. It causes more problems than it solves in dense areas because all channels may experience high interference, and it will switch, switch and switch. Only permit channel switching during a allowed time range or with 0 clients connected. Yes when using DFS channels it can occasionally experience a channel change if it detects radar, at that there is even a protocol to notify the clients of the change. Especially with the dual 5G you have going, you really need to enable DFS. In my professional opinion your making things harder for yourself and your users.

Re: Radio Profile - channel selection auto - neighbor APs on same channel & running at Max power allowed.

w1f1n00b — Wed, 25 Nov 2020 00:51:58 GMT

The problem I’m having is that APs don’t appear to respect the DCS settings. I tested this, and GTAC confirmed, that even with DCS disabled, APs will still change channels based on interference. This is why they recommended the route of manually adjusting radio powers etc. (however I think there is more going on than that)

I set up three test sites

site 1-DCS OFF - still seeing channel changes

site 2- DCS ON, Interference Threshold OFF -still seeing channel changes outside of maintenance window (which it should only due based on interference, which based on policy wasn’t allowed)

site 3- DCS ON, Interference Threshold ON - many channel changes UNTIL… I applied a beta firmware on 11/5 due to an open ticket regarding high crc errors, and since then all channel changes have been inside the scheduled maintenance window.

So. their might be light at the end of this tunnel. I just wish there was better documentation of ongoing issues so I don’t have to beat the bushes trying to figure this stuff out.

@Sam Pirok do you have any info regarding acsp, background scans, DCS, and DFS channels? I’d love to get some clarification since I was told by GTAC while on the phone with my SE working on a ticket, that APs will not perform background scanning while using DFS channels.

Re: Radio Profile - channel selection auto - neighbor APs on same channel & running at Max power allowed.

jose_gonzalez — Wed, 25 Nov 2020 01:32:59 GMT

Re: Radio Profile - channel selection auto - neighbor APs on same channel & running at Max power allowed.

jose_gonzalez — Wed, 25 Nov 2020 01:40:15 GMT

What firmware code, AP model(s) 250/650 are you seeing this DCS issue? What is the development build firmware version from ‘show ver’?

Re: Radio Profile - channel selection auto - neighbor APs on same channel & running at Max power allowed.

w1f1n00b — Wed, 25 Nov 2020 04:21:05 GMT

from the sites I mentioned above, they’re all AP250, however I’m seeing similar issues with AP550 as well. This has been ongoing for at least 8 months (the false high interference even longer, I think a year+)

Site 1 -HiveOS 10.0r9a build-247125 (issue has been present through several firmware versions)

Site 2 -HiveOS 10.0r9a build-247125 (issue has been present through several firmware versions)

Site 3 -HiveOS 10.0r9a Nantucket build-250402-slnu-dev

Re: Radio Profile - channel selection auto - neighbor APs on same channel & running at Max power allowed.

daniel1 — Wed, 25 Nov 2020 18:01:56 GMT

At least for the high inference there should be a fix incoming: https://extremeportal.force.com/ExtrArticleDetail?n=000054239

Re: Radio Profile - channel selection auto - neighbor APs on same channel & running at Max power allowed.

w1f1n00b — Wed, 25 Nov 2020 23:00:15 GMT

@daniel.schickmair , thanks for sharing that. I hadn’t seen that document yet, but had been told this was coming. How did you find this doc? I’d love to stay on top of these types of posts so I’m in the loop. I’ve poked around the KB and Tech Docs but it’s tricky to navigate and I haven’t been able to find that doc aside from your link.

I’m also curious if this fix is going to have any affect on ACSP or DCS, as that is as big of a concern to me as Interference Alarms.

Re: Radio Profile - channel selection auto - neighbor APs on same channel & running at Max power allowed.

daniel1 — Wed, 25 Nov 2020 23:18:56 GMT

I’m periodically checking the Extreme Portal for new articles for our AP650 (because as you know, known issues in the release notes are not reliable (yet) ).

Just input in your case “AP250” or your current firmware version “10.0r9a” (with double quotes) in the search field, filter Knowledge as source on the left and sort after date.

Re: Radio Profile - channel selection auto - neighbor APs on same channel & running at Max power allowed.

w1f1n00b — Thu, 26 Nov 2020 02:38:54 GMT

Here’s another example I just came across where it appears ACSP is struggling. I find it interesting that only the APs on ch36 have adjusted radio power down, while all other radios are running at max allowed power (20dB in this case). I will admit, this device template needs tweaking as it’s meant for lower density deployments, and this site has a higher than typical number of APs, but it does provide a good example of what appear to be the current limitations or deficiencies with ACSP. ACSP Neighbor CLI is from AP in 03-005

Re: Radio Profile - channel selection auto - neighbor APs on same channel & running at Max power allowed.

jose_gonzalez — Thu, 26 Nov 2020 11:11:05 GMT

@john_kern In your heatmap screenshot you have 10 APs, if both radios are in 5ghz that’s 20 channels. Looking at your ACSP channel cost screenshot you have 4 channels on the low side of the 5ghz band and 5 on the high side, for a total of 9. So for that small area you have 9 available channels for 20 broadcasts. ACSP is doing a bad job because there is so much overlap. Enable DFS and ACSP will do a better job.