Wireless clients sporadically getting Limited Connectivity, APs dropping packets

[John_Rowler]

We have a pair of V2110 wireless controllers and around 400 3825i access points. Controller firmware is 09.21.06.0002. AP firmware is 9.21.27.1387X.

About 4 months ago we noticed a problem where the APs would just drop traffic. A packet capture will show a ping coming into the radio from the client, going out the Eth interface on the AP, the reply coming back into the Eth interface, and never going out of the radio.

A particularly strange detail is that this only happens with certain source/destination traffic. For instance a client cant ping the gateway of 10.1.20.1/24. but cannot pint the server at 10.1.20.2. Since we can see the ping making it to the server and the ping reply making it back to the AP, we know it's not a firewall/ACL/routing issue on any other part of the network. The traffic disappears at the AP. Clients cannot communicate with anything on the Internet or any of our servers (DNS, DHCP, etc.) during this time. They can ping gateways in any subnet but that's it; everything else is failed.

More details:

-We've gone through several firmware updates with GTAC. Nothing has helped.
-The issue can affect any client on any AP in any location at any time.
-The issue is intermittent; it may happen every few minutes or once per month to a particular client in a particular area. Once failed it may last for seconds or days.
-While it's affecting one client, other clients on the AP may be working fine or may also fail.
-It can happen on either 2.4 or 5.7GHz.
-It can affect any type of client (phone, laptop, tablet) and any OS.
-Restarting the client or AP will sometimes fix the issue, sometimes not.
-It happens with our without using NAC, or any other type of authentication. It happens on completely open networks as well.
-It happens on both bridged at AP and bridged at controller topologies.
-It's not limited to a particular subnet or VLAN.
-It happens with or without Flexible Client Access enabled.
-It happens on either controller.
-GTAC hasn't made much progress on the issue in 4 months.

[FES]

update: in 15 minutes we have had 2 dissconnections. In iphone 5s client and in mac book air. In 10-20 secconds connect again.

[FES]

Thanks Raffi,
we'll try to capture logs to identify the problem.
Now the dessconection is in our laptop mac book air (capitan osx), with no roaming and no movement of the device.
The ap dont have more than 10 users. Otherwise, the band select doesnt work fine in some apple devices... theirs enter in a stuck mode and dont connect in 2,4 neither 5 band.

[Raffi]

Troubleshooting intermittent issues is always difficult.

A suggestion is to get to users who are having issues, and to also actively go about the environment with your own equipment, using the network with your own equipment to hopefully catch something happening. Once the opportunity presents itself to troubleshoot, then things to look for is to search for the problem client's mac on the controller gui reports, like reports->wireless clients by ap. Take a screenshot, it will show you what ap you are on, what signal the ap sees the client at, what radio you are on. Then you can navigate to that ap under the wireless aps section, and enable remote capture. You can also get the aps current ip from the static tab under the aps config. Then you can point your wireshark and do a capture on wifi0 for 5ghz radio, wifi1 for 2.4ghz radio, and eth0 for the ap's wired Ethernet port. If you have time, best to capture the wireless and wired quickly while the client is doing something like pinging the default gateway, and something off the network like google, 8.8.8.8.

You could also check for any messages for the client mac under controller->logs->station events, pasting the client mac in the search window.

Good to also collect the ap trace file, log->ap traces->select ap and retrieve trace.

Check the client and verify that its drivers are updated and windows power management for the wireless interface is off for maximum performance.

Verify if all users on the ap are having issues, or just one client.

Also, under wlan, make sure the the pre/post and session timeouts make sense ( ie pre=5 post=30 and session=0 ), otherwise you'll have ton's of user sessions that are stale on the controller.

Use radio preference group to push 5ghz capable clients to use 5ghz, a much better band with more channels and less interference.

It can turn out that in 10 complaints, each case is different...ie one user connected hanging on to a far away ap with poor signal because the client decides not to roam to an ap closer. One case could be a client with pwr mgmt. enabled. One trace could show the client probe req, and probe rsp coming back but the client not proceeding to associate, etc. On the surface they would all look the same.

[Andre_Brits_Kan]

Hi All

I have just stumbled across this post.
We are experiencing exactly the same issues as described from the beginning of this post.

We are running 3825's on a V2110 controller with the new version 10 code.
Random users would get limited connectivity, only when they disconnect and reconnect do they start transferring traffic.
I have not done a wireshark trace but will do so.
What I have noticed for the users that complain about this problem is that in the client report under the "Avg.Rate(Mbps) Sent/Rec'd" column the sent rate is 0 but the Rec'd rate displays a value.

I will report back on the wireshark trace.

Thx

[FES]

Hi JP,
we have the same configuration. Radius-Ldap, in the user file add the Session-timeout value.

To test if is working fine, run radius in debug mode. radius -X. In the Access-Accept message you must see the value.

Dought the 9.21.08 firmware doestn solve the problem. We have installed de 08 versión the last sunday... 4 days and the behaviour is the same. We have improve the reconnect with the session timeout option, but we have this bad behaviour in some devices. At now, we have problems with mac devices... the device lost the conexion and in 20 seconds reconnect again..

Know we have session timeout value in controller in 0.