topic Re: CAPWAP failures occurring after a move to hive connect (free) from the hive select (paid) in Aerohive Migrated Content

CAPWAP failures occurring after a move to hive connect (free) from the hive select (paid)

art_obrien — Wed, 13 Nov 2019 00:06:50 GMT

We recently moved some AP250 APs to cloud connect (free) from the paid cloud to test that platform before deciding on license renewal. Settings between both platforms were duplicated. We're seeing APs loose connection to the hive yet internal monitoring shows no loss of connection from the network by ping.

I've worked through the steps listed in this help document.

https://thehivecommunity.aerohive.com/s/article/CAPWAP-Guide-for-HiveManager-NG

The APs have been set to capwap over http on 80. I've turned on capwap debug for a particular AP and its output is below. For time reference, the hive reported a loss of connection via email alert at 1:46PM.

All help appreciated.

Re: CAPWAP failures occurring after a move to hive connect (free) from the hive select (paid)

AnonymousM — Wed, 13 Nov 2019 05:14:37 GMT

Hey Art,

The log does show a disconnect around 13:40, but this log doesn't indicate what caused it outside of the ah_capwap_event_timer timed out. Capwap is latency sensitive and occasionally picked up by firewalls with IDS/IPS rules enabled because of the nature of the protocol. There could also be some capwap settings from the original HM Select account like the VHM-ID that might be causing issues.

Run #show capwap client /*and note the VHM-ID compared to your HM Connect Instance

CAPWAP client: Enabled

CAPWAP transport mode: HTTP on TCP

RUN state: Connected securely to the CAPWAP server

CAPWAP client IP: xxx.xxx.xxx.xxx

CAPWAP server IP: 34.202.197.42

HiveManager Primary Name:cloud-va2-cws-10.aerohive.com

HiveManager Backup Name: hmng-prod-va2-cm-01.aerohive.com

CAPWAP Default Server Name: redirector.aerohive.com

Virtual HiveManager Name: VHM-MDYREIVR

2019-11-11 13:40:52 debug capwap: [capwap_info]: ah_capwap_event_timer timed out

2019-11-11 13:40:52 debug capwap: [capwap_basic]: current parameters:DTLS cut DTLS abort DTLS_delete_timer

2019-11-11 13:40:52 debug capwap: [capwap_info]: DTLS ABORT->Enter the DTLS abort State.

2019-11-11 13:40:52 debug capwap: [capwap_info]: set timer type is DTLS_delete_timer interval is 5

2019-11-11 13:40:52 debug capwap: [capwap_info]: DTLS ABORT->Leave the DTLS abort State.

2019-11-11 13:40:52 debug capwap: [capwap_basic]: capwap set watch dog:125 priority:5 modid:4

2019-11-11 13:40:52 debug capwap: [capwap_basic]: CAPWAP: cur_status:DTLS cut, cur_event:DTLS abort, timer to fire:5

2019-11-11 13:40:57 debug capwap: [capwap_info]: timer ah_capwap_dtls_disconn_timer timed out

2019-11-11 13:40:57 debug capwap: [capwap_basic]: current parameters:Start waitting for cli none

2019-11-11 13:40:57 debug capwap: [capwap_info]: START->Enter the Start State.

2019-11-11 13:41:09 debug ah_dcd: b3300618 77 00 00 00 3a 04 d3 cf e8 0f bf f4 ea b5 07 5d w...:... .......]

2019-11-11 13:41:09 debug ah_dcd: b3300628 54 0b 00 00 01 00 00 00 26 00 02 01 0d 01 01 01 T....... &.......

2019-11-11 13:41:09 debug ah_dcd: b3300638 02 01 03 03 01 06 04 02 02 12 05 01 00 06 01 01 ........ ........

2019-11-11 13:41:09 debug ah_dcd: b3300648 07 01 03 08 01 01 09 01 00 0a 01 00 0b 01 00 ........ .......

2019-11-11 13:41:14 debug capwap: [capwap_info]: unregister port:80, proxy port:80

2019-11-11 13:41:14 debug capwap: [capwap_basic]: state: Start--->Waitting for HiveAP IP, event: waitting for cli--->none

2019-11-11 13:41:14 debug capwap: [capwap_info]: HTTP proxy done its first time connecting to HM

2019-11-11 13:41:14 debug capwap: [capwap_info]: capwap handle_bonjour_service service old 0 new 2

2019-11-11 13:41:14 debug capwap: [capwap_info]: START->Leave the Start State.

2019-11-11 13:41:14 debug capwap: [capwap_basic]: capwap set watch dog:60 priority:5 modid:4

2019-11-11 13:41:14 debug capwap: [capwap_basic]: current parameters:Waitting for HiveAP IP none none

2019-11-11 13:41:14 debug capwap: [capwap_info]: START->Enter the Get Host IP State.

2019-11-11 13:41:14 debug capwap: [capwap_info]: device ip is 10.109.35.140

2019-11-11 13:41:14 debug capwap: [capwap_info]: unregister port:80, proxy port:80

Re: CAPWAP failures occurring after a move to hive connect (free) from the hive select (paid)

samantha_lynn — Wed, 13 Nov 2019 06:12:53 GMT

Thank you for that data. We're seeing a lot of CAPWAP timers failing, for instance:

capwap: [capwap_info]: ah_capwap_discovey_timer timed out

capwap: [capwap_info]: ah_capwap_maxdisco_timer timed out

capwap: [capwap_info]: ah_capwap_idle_timer timed out

capwap: [capwap_info]: timer ah_capwap_dtls_disconn_timer timed out

capwap: [capwap_info]: ah_capwap_event_timer timed out

This indicates an issue with latency on the backend network hosting the APs. If you can send a packet capture we can help check for things like individual clients flooding the network and an overload of multicast or broadcast traffic.

Additionally, could you verify that you have the following ports allowing outbound traffic on your firewall:

UDP 12222

TCP 443

TCP 22

Re: CAPWAP failures occurring after a move to hive connect (free) from the hive select (paid)

art_obrien — Thu, 14 Nov 2019 03:49:22 GMT

Yes we have verified the correct vhm and capwap address for the new hive along with the necessary ports. We've also full reset a few aps and are still seeing the issue. The issue wasn't occurring prior to the hive change so I don't see how multicast/mdns would be different between the hives. They have been in stable production for some time until the hive change.

Attached is a packet capture of a particular AP. Failure was reported by email at 2:30PM. I didn't see anything especially obvious in the communication between the AP (10.109.35.123) and the capwap server (34.202.197.49). Appologies for the long capture while waiting for a failure.

Re: CAPWAP failures occurring after a move to hive connect (free) from the hive select (paid)

samantha_lynn — Thu, 14 Nov 2019 22:32:39 GMT

Thank you for that packet capture. The good news is I don't see any particular clients that are sending too much traffic. The bad news is, 90.9% of your traffic is multicast or broadcast traffic (you can see this with the filter "(eth.dst[0]&1)" ). That is a very high percentage of your traffic, and too much multicast traffic has been known to cause enough delay that it interrupts the CAPWAP process. Would you be able to limit the amount of multicast allowed through the network?

Re: CAPWAP failures occurring after a move to hive connect (free) from the hive select (paid)

art_obrien — Thu, 14 Nov 2019 23:40:48 GMT

Why are the same APs in the same production environment not seeing the same issue while in the paid hive manager?

Re: CAPWAP failures occurring after a move to hive connect (free) from the hive select (paid)

AnonymousM — Thu, 14 Nov 2019 23:47:03 GMT

Without reference to physical location, network topology, and AP configurations it's hard to tell. Do you mind at least sharing the running configs from an AP in Select and an AP in Connect to see if there is a basic configuration difference between the two?

Re: CAPWAP failures occurring after a move to hive connect (free) from the hive select (paid)

art_obrien — Fri, 15 Nov 2019 23:29:57 GMT

This one is still in the free hive.

Re: CAPWAP failures occurring after a move to hive connect (free) from the hive select (paid)

art_obrien — Fri, 15 Nov 2019 23:31:42 GMT

This one was one of our most frequent failures. I moved it back to the paid hive last night and it hasn't dropped again. There are a few differences but nothingI noticed capwap related.