Wireless clients sporadically getting Limited Connectivity, APs dropping packets

[John_Rowler]

We have a pair of V2110 wireless controllers and around 400 3825i access points. Controller firmware is 09.21.06.0002. AP firmware is 9.21.27.1387X.

About 4 months ago we noticed a problem where the APs would just drop traffic. A packet capture will show a ping coming into the radio from the client, going out the Eth interface on the AP, the reply coming back into the Eth interface, and never going out of the radio.

A particularly strange detail is that this only happens with certain source/destination traffic. For instance a client cant ping the gateway of 10.1.20.1/24. but cannot pint the server at 10.1.20.2. Since we can see the ping making it to the server and the ping reply making it back to the AP, we know it's not a firewall/ACL/routing issue on any other part of the network. The traffic disappears at the AP. Clients cannot communicate with anything on the Internet or any of our servers (DNS, DHCP, etc.) during this time. They can ping gateways in any subnet but that's it; everything else is failed.

More details:

-We've gone through several firmware updates with GTAC. Nothing has helped.
-The issue can affect any client on any AP in any location at any time.
-The issue is intermittent; it may happen every few minutes or once per month to a particular client in a particular area. Once failed it may last for seconds or days.
-While it's affecting one client, other clients on the AP may be working fine or may also fail.
-It can happen on either 2.4 or 5.7GHz.
-It can affect any type of client (phone, laptop, tablet) and any OS.
-Restarting the client or AP will sometimes fix the issue, sometimes not.
-It happens with our without using NAC, or any other type of authentication. It happens on completely open networks as well.
-It happens on both bridged at AP and bridged at controller topologies.
-It's not limited to a particular subnet or VLAN.
-It happens with or without Flexible Client Access enabled.
-It happens on either controller.
-GTAC hasn't made much progress on the issue in 4 months.

[James_A]

What version of 9.15 did you try? I loaded 9.15.10 firmware onto a few APs, but still had the problem.

Good luck with the escalation, we've had our cases escalated twice and *crickets*. Even after sending in some excellent packet captures showing the AP unable to decode some packets from the client, and associated log entries in the AP trace. I still need to do a monitor mode capture on a client to confirm it can see the missing ARP packets coming over the air.

Also, it's radio dependent - I've had the problem on 5GHz, moved to 2.4GHz on the same AP and there's no problem. It can happen anywhere, but some APs are more prone to it it seems.

[Ronald_Dvorak]

I'm not sure whether I've run into the same issue today but just in case I'd like to share how we've fixed it (thanks for the great support from the GTAC@Ryan).

Clients on a AP3805i with bridge@AP topology and 802.1X PEAP were not able to rx/tx data.
The MAC of the client wasn't learned on the switchport the client/AP was connected to - the client/AP report listed the client as connected and authenticated.
Only some types of laptops were affected - I don't think that any mobiles had issues.

After we've disabled "ARP proxy" on the bridge@AP topology everything was OK.

I've done the same for another new installation where some clients were stuck with IP 169.x.x.x and that is also working now.
Software in use was 9.21.08 (customer#1) and 10.01.04.0011 (customer#2).

GTAC ticket# 01212080

-Ron

[Chat]

Later, when you disable wifi's laptop and enable again client's MAC show on switch port.

[Chat]

Confirmed, when client fail, client's MAC not show on switchport.

[Chat]

Hi Ronald,
We have the same scenario and disabled "ARP proxy" on the bridge@AP topology, but it doesn't work.
(2) V2110 in HA.
fw. 09.21.08
APs 3825/3805/3865

We are trying to see MAC client learned on the switchport when clients fail.

We'll tell you something later.