Wireless clients sporadically getting Limited Connectivity, APs dropping packets

[John_Rowler]

We have a pair of V2110 wireless controllers and around 400 3825i access points. Controller firmware is 09.21.06.0002. AP firmware is 9.21.27.1387X.

About 4 months ago we noticed a problem where the APs would just drop traffic. A packet capture will show a ping coming into the radio from the client, going out the Eth interface on the AP, the reply coming back into the Eth interface, and never going out of the radio.

A particularly strange detail is that this only happens with certain source/destination traffic. For instance a client cant ping the gateway of 10.1.20.1/24. but cannot pint the server at 10.1.20.2. Since we can see the ping making it to the server and the ping reply making it back to the AP, we know it's not a firewall/ACL/routing issue on any other part of the network. The traffic disappears at the AP. Clients cannot communicate with anything on the Internet or any of our servers (DNS, DHCP, etc.) during this time. They can ping gateways in any subnet but that's it; everything else is failed.

More details:

-We've gone through several firmware updates with GTAC. Nothing has helped.
-The issue can affect any client on any AP in any location at any time.
-The issue is intermittent; it may happen every few minutes or once per month to a particular client in a particular area. Once failed it may last for seconds or days.
-While it's affecting one client, other clients on the AP may be working fine or may also fail.
-It can happen on either 2.4 or 5.7GHz.
-It can affect any type of client (phone, laptop, tablet) and any OS.
-Restarting the client or AP will sometimes fix the issue, sometimes not.
-It happens with our without using NAC, or any other type of authentication. It happens on completely open networks as well.
-It happens on both bridged at AP and bridged at controller topologies.
-It's not limited to a particular subnet or VLAN.
-It happens with or without Flexible Client Access enabled.
-It happens on either controller.
-GTAC hasn't made much progress on the issue in 4 months.

[Doug]

Yes, I would open a ticket for tracking.

-Doug

[Andre_Brits_Kan]

Hi Doug

Case Number: 01206949

Thx

[Andre_Brits_Kan]

Hi All

Just to give some feedback:

Firstly to summarize my scenario:

Controller = V2110

Code = 10.01.01

AP Model = 3825i

Setup:

SSID with Preshared key SSID with MAC auth to NAC

Problems Seen:

Client not being able to connect Clients randomly disconnected every few minutes Very slow response

See reference Client connection issues with firmware 9.21.07 or 10.01.03

As a work around I have deployed a second V2110 controller running V9.15 (Eval License)

I have migrated the AP's across from the V10 controller to the V9 controller.

The client is currently 100% happy, all the users can connect that could not in the past and none of the connections is dropped as was seen every few minutes.

@Doug

I have not logged a case with GTAC for this, would you like me to open an official case?

Regards

[JP4]

Were you able to re-create the problem easily with your devices at the location before going back to the earlier 9.15 code ?

[Andre_Brits_Kan]

Currently we are running a V2110 controller with V10.01.01.0129 and see the "client disconnects" and "unable to connect clients".
As a work around I have deployed another V2110 controller running a evaluation Licenses.
This new controller is running the older V9.15.10 (The first version to support the 3825 AP's).

I have migrated the AP's from the V10 controller to the V9 controller.
The AP's downgraded and is operating on the old Code.

All the users can now connect, even the ones that could not connect before.
I also have no more complaints about dropped connections.

I will run the system like this until the updated V10 code is release.
Once I have upgraded the V10 controller I will migrate a few AP's back and test accordingly.

Thx