10.2r4 firmware causing AP650's to lose connectivity?

[KevinP1]

Okay, I don’t know what’s happening here but this isn’t looking pretty. All of our AP650’s are losing connectivity multiple times throughout the day. 

Look at this! Some AP’s will only do this once or twice and others are doing it often, like this ? There’s nothing indicating any type of error or issue. We didn’t have this problem on firmware 10.09rb.

I have a case open with GTAC, but are we the only ones seeing this issue?

[Matt_King]

Hi John,

Were having very similar issues here. we predominantly have AP130’s and 230’s which have been running 10.0r9b although I have tried the firmware you suggested and others on this post but am still getting complaints. Just like you mention users appear to be connected but are unable to access the internet. 

I was interested in your post above, could you confirm if these features were disabled or enabled when you started to notice a difference?  Currently I have some of these still enabled on our AP’s

Thanks in advance,

Matt.

[KevinP1]

@javabomberman @systemscsn 

The roller coaster waves have greatly diminished. I have some AP’s that look great and some that look like this AP ?

After extensive syslog’s, packet captures, and everything else under the sun; we have not been able to figure out why we are seeing these dips in XIQ. However, at this point, the dips do not currently coenside with client drops or performance decline. We are currently on 10.3r1 and it’s probably the most stable firmware we’ve seen, thankfully. I will check back in with our engineer to see if the XIQ department knows exactly why this is happening. Also, if you’re able to, upgrade up to 10.3r1 because you’ll see things stablize quickly.

[systemscsn]

So @kevin.piazza any update?

[Hans_SD44]

@kevin.piazza @Sam Pirok 

How are things looking now? I’m still wondering what correlation there is, if any, between actual disconnects and the memory dumps shown in the graphs. I’m still seeing dips on 10.09rb, but can’t really pinpoint if they are tied to disconnects or if the sparse reports we get are connected to AP issues at all. (as is the constant battle in a school system.)

[KevinP1]

@john_kern You’ve been having a jolly good ‘ole time! We are still seeing the graph dips as well, and are syslogging one AP to hopefully figure out that issue. However, we have reached a level of stability at this point. I’ve been working with GTAC, Chris (he’s been awesome!), for hours and hours with identifying the issues; there were plenty of them.

All of our AP’s had been rolled back to 10.0.9rb. There were AP’s with errors in the flash logs, buff logs, etc. Here are the steps we had to take to prevent clients from dropping and having healthy connections:

1. The first thing we did was disable WIPS, since in older firmware there’s more of a possibility of this having complications to a certain extent. 
2. Disabled Weak Signal Probe Request Suppression
3. Upgraded all AP’s to 10.3r1
4. Changed 5GHz radio from 20MHz to 40MHz channel width (20MHz was too noisy even though it’s been stated that “This is the Way”  for having an AP in every classroom and multiple floors)
5. Adjusted some AP power levels to accommodate the channel width change
6. Turned on “Enable short guard interval”
7. Currently testing with WIPS turned back on with manual remediation
8. Added the MAC OUI’s into the WIPS list (only had the SSID’s previously)

We’ll see how this goes but WOW this taken some time to narrow down. We’re still isolating a few small things like we had mentioned. I’ll keep everyone updated.