CAPWAP failures occurring after a move to hive connect (free) from the hive select (paid)

[art_obrien]

We recently moved some AP250 APs to cloud connect (free) from the paid cloud to test that platform before deciding on license renewal. Settings between both platforms were duplicated. We're seeing APs loose connection to the hive yet internal monitoring shows no loss of connection from the network by ping.

I've worked through the steps listed in this help document.

https://thehivecommunity.aerohive.com/s/article/CAPWAP-Guide-for-HiveManager-NG

The APs have been set to capwap over http on 80. I've turned on capwap debug for a particular AP and its output is below. For time reference, the hive reported a loss of connection via email alert at 1:46PM.

All help appreciated.

[samantha_lynn]

Thank you for that packet capture. The good news is I don't see any particular clients that are sending too much traffic. The bad news is, 90.9% of your traffic is multicast or broadcast traffic (you can see this with the filter "(eth.dst[0]&1)" ). That is a very high percentage of your traffic, and too much multicast traffic has been known to cause enough delay that it interrupts the CAPWAP process. Would you be able to limit the amount of multicast allowed through the network?

[art_obrien]

Yes we have verified the correct vhm and capwap address for the new hive along with the necessary ports. We've also full reset a few aps and are still seeing the issue. The issue wasn't occurring prior to the hive change so I don't see how multicast/mdns would be different between the hives. They have been in stable production for some time until the hive change.

Attached is a packet capture of a particular AP. Failure was reported by email at 2:30PM. I didn't see anything especially obvious in the communication between the AP (10.109.35.123) and the capwap server (34.202.197.49). Appologies for the long capture while waiting for a failure.

[samantha_lynn]

Thank you for that data. We're seeing a lot of CAPWAP timers failing, for instance:

capwap: [capwap_info]: ah_capwap_discovey_timer timed out

capwap: [capwap_info]: ah_capwap_maxdisco_timer timed out

capwap: [capwap_info]: ah_capwap_idle_timer timed out

capwap: [capwap_info]: timer ah_capwap_dtls_disconn_timer timed out

capwap: [capwap_info]: ah_capwap_event_timer timed out

This indicates an issue with latency on the backend network hosting the APs. If you can send a packet capture we can help check for things like individual clients flooding the network and an overload of multicast or broadcast traffic. 

Additionally, could you verify that you have the following ports allowing outbound traffic on your firewall:

UDP 12222

TCP 443

TCP 22

[AnonymousM]

Hey Art,

The log does show a disconnect around 13:40, but this log doesn't indicate what caused it outside of the ah_capwap_event_timer timed out. Capwap is latency sensitive and occasionally picked up by firewalls with IDS/IPS rules enabled because of the nature of the protocol. There could also be some capwap settings from the original HM Select account like the VHM-ID that might be causing issues.

Run #show capwap client /*and note the VHM-ID compared to your HM Connect Instance

CAPWAP client: Enabled

CAPWAP transport mode: HTTP on TCP

RUN state: Connected securely to the CAPWAP server

CAPWAP client IP: xxx.xxx.xxx.xxx

CAPWAP server IP: 34.202.197.42

HiveManager Primary Name:cloud-va2-cws-10.aerohive.com

HiveManager Backup Name: hmng-prod-va2-cm-01.aerohive.com

CAPWAP Default Server Name: redirector.aerohive.com

Virtual HiveManager Name: VHM-MDYREIVR

2019-11-11 13:40:52 debug  capwap: [capwap_info]: ah_capwap_event_timer timed out

2019-11-11 13:40:52 debug  capwap: [capwap_basic]: current parameters:DTLS cut DTLS abort DTLS_delete_timer

2019-11-11 13:40:52 debug  capwap: [capwap_info]: DTLS ABORT->Enter the DTLS abort State.

2019-11-11 13:40:52 debug  capwap: [capwap_info]: set timer type is DTLS_delete_timer interval is 5

2019-11-11 13:40:52 debug  capwap: [capwap_info]: DTLS ABORT->Leave the DTLS abort State.

2019-11-11 13:40:52 debug  capwap: [capwap_basic]: capwap set watch dog:125 priority:5 modid:4

2019-11-11 13:40:52 debug  capwap: [capwap_basic]: CAPWAP: cur_status:DTLS cut, cur_event:DTLS abort, timer to fire:5

2019-11-11 13:40:57 debug  capwap: [capwap_info]: timer ah_capwap_dtls_disconn_timer timed out

2019-11-11 13:40:57 debug  capwap: [capwap_basic]: current parameters:Start waitting for cli none

2019-11-11 13:40:57 debug  capwap: [capwap_info]: START->Enter the Start State.

2019-11-11 13:41:09 debug  ah_dcd: b3300618 77 00 00 00 3a 04 d3 cf e8 0f bf f4 ea b5 07 5d  w...:... .......]

2019-11-11 13:41:09 debug  ah_dcd: b3300628 54 0b 00 00 01 00 00 00 26 00 02 01 0d 01 01 01  T....... &.......

2019-11-11 13:41:09 debug  ah_dcd: b3300638 02 01 03 03 01 06 04 02 02 12 05 01 00 06 01 01  ........ ........

2019-11-11 13:41:09 debug  ah_dcd: b3300648 07 01 03 08 01 01 09 01 00 0a 01 00 0b 01 00    ........ .......

2019-11-11 13:41:14 debug  capwap: [capwap_info]: unregister port:80, proxy port:80

2019-11-11 13:41:14 debug  capwap: [capwap_basic]: state: Start--->Waitting for HiveAP IP, event: waitting for cli--->none

2019-11-11 13:41:14 debug  capwap: [capwap_info]: HTTP proxy done its first time connecting to HM

2019-11-11 13:41:14 debug  capwap: [capwap_info]: capwap handle_bonjour_service service old 0 new 2

2019-11-11 13:41:14 debug  capwap: [capwap_info]: START->Leave the Start State.

2019-11-11 13:41:14 debug  capwap: [capwap_basic]: capwap set watch dog:60 priority:5 modid:4

2019-11-11 13:41:14 debug  capwap: [capwap_basic]: current parameters:Waitting for HiveAP IP none none

2019-11-11 13:41:14 debug  capwap: [capwap_info]: START->Enter the Get Host IP State.

2019-11-11 13:41:14 debug  capwap: [capwap_info]: device ip is 10.109.35.140

2019-11-11 13:41:14 debug  capwap: [capwap_info]: unregister port:80, proxy port:80