Wireless clients sporadically getting Limited Connectivity, APs dropping packets

[John_Rowler]

We have a pair of V2110 wireless controllers and around 400 3825i access points. Controller firmware is 09.21.06.0002. AP firmware is 9.21.27.1387X.

About 4 months ago we noticed a problem where the APs would just drop traffic. A packet capture will show a ping coming into the radio from the client, going out the Eth interface on the AP, the reply coming back into the Eth interface, and never going out of the radio.

A particularly strange detail is that this only happens with certain source/destination traffic. For instance a client cant ping the gateway of 10.1.20.1/24. but cannot pint the server at 10.1.20.2. Since we can see the ping making it to the server and the ping reply making it back to the AP, we know it's not a firewall/ACL/routing issue on any other part of the network. The traffic disappears at the AP. Clients cannot communicate with anything on the Internet or any of our servers (DNS, DHCP, etc.) during this time. They can ping gateways in any subnet but that's it; everything else is failed.

More details:

-We've gone through several firmware updates with GTAC. Nothing has helped.
-The issue can affect any client on any AP in any location at any time.
-The issue is intermittent; it may happen every few minutes or once per month to a particular client in a particular area. Once failed it may last for seconds or days.
-While it's affecting one client, other clients on the AP may be working fine or may also fail.
-It can happen on either 2.4 or 5.7GHz.
-It can affect any type of client (phone, laptop, tablet) and any OS.
-Restarting the client or AP will sometimes fix the issue, sometimes not.
-It happens with our without using NAC, or any other type of authentication. It happens on completely open networks as well.
-It happens on both bridged at AP and bridged at controller topologies.
-It's not limited to a particular subnet or VLAN.
-It happens with or without Flexible Client Access enabled.
-It happens on either controller.
-GTAC hasn't made much progress on the issue in 4 months.

[JP4]

I have not done 9.21.08 yet, working on getting that scheduled in a maintenance window. GTAC had gotten us to an improved state with some config changes and code just prior to 9.21.08 Some of the outstanding issues could have been caused by the probe suppression also.

[FES]

if the client don't have de certificate or de correct configuration we can see a TLS error en radius log and the client can't connect.
Have you upgraded to 9.21.08 version?

[JP4]

Thanks for the info. When you are having clients with cert problems, do they get connected at all ?

[FES]

JP, have you checked if you have Probe Suppression Enabled? 9.21.08 release solve this problem.
I am in 9.21.07 release with Probe Suppression Disable. I have improve the problems with iPhones enabling the WPA2 key management options.
Today I have some problems in one MacBookPro with reconnecting. Sometimes the reconnecting was solved disabling Frame Transition (802.11r). WPA1 is disabled.
With band select enabled in the ap I have problems with iOS an Macosx and i have deletes this option.

Also, have you checked that the clients have the supplicant correctly installed, I mean, that you have installed the certificated of the radius server in the clients. I have had a lot of cases that the problem have that incorrect configuration of the supplicant.

Otherwise, when i have upgrade my mac from yosemite to capitan the 802.1x profile is broken and you have to delete and install again.

I hope this helps you.

[JP4]

Ok thanks. I was going to use the multi-edit, but wanted to see if I could get an estimate of how many had it enabled before hand.