topic Re: Device AH-xxxxxx Connect State Change: The CAPWAP connection was lost. in Aerohive Migrated Content

Device AH-xxxxxx Connect State Change: The CAPWAP connection was lost.

stewart_mahoney — Wed, 08 May 2019 16:05:04 GMT

Device AH-xxxxxx Connect State Change: The CAPWAP connection was lost.

I am getting the above error happen every 10-15 minutes on each of my connected APs. They all seem to work in that I can browse out onto the Internet, and have access to all relevant network resources when connected to them. All relevant traffic is allowed via our firewall.

Below are the logs I have found around the time it last happened:

2019-05-08 09:17:50 info amrp2: l2routing: amrp receive capwap-statistic-request Mod_id (4) No.(183) ifindex 9

2019-05-08 09:17:49 info capwap: application: CAPWAP: can not find the event query confirm information type request id (44).

2019-05-08 09:17:49 info capwap: application: CAPWAP: can not find the event request id (39).

2019-05-08 09:17:49 info capwap: application: CAPWAP: can not find the event query confirm information type request id (39).

2019-05-08 09:17:49 info capwap: application: CAPWAP: can not find the event request id (40).

2019-05-08 09:17:49 info capwap: application: CAPWAP: can not find the event query confirm information type request id (40).

2019-05-08 09:17:48 info capwap: application: CAPWAP: receive DCD send response to CAPWAP event!, length:570

2019-05-08 09:17:48 info capwap: application: receive event DCD send response to CAPWAP: eventid = 169: length = 570

2019-05-08 09:17:48 info capwap: application: CAPWAP: receive DCD send response to CAPWAP event!, length:1308

2019-05-08 09:17:48 info capwap: application: receive event DCD send response to CAPWAP: eventid = 169: length = 1308

2019-05-08 09:17:48 info capwap: application: CAPWAP: receive DCD send response to CAPWAP event!, length:38

2019-05-08 09:17:48 info capwap: application: receive event DCD send response to CAPWAP: eventid = 169: length = 38

2019-05-08 09:17:48 info capwap: application: CAPWAP: receive DCD send response to CAPWAP event!, length:146

2019-05-08 09:17:48 info capwap: application: receive event DCD send response to CAPWAP: eventid = 169: length = 146

2019-05-08 09:17:48 info capwap: application: CAPWAP: receive DCD send response to CAPWAP event!, length:336

2019-05-08 09:17:48 info capwap: application: receive event DCD send response to CAPWAP: eventid = 169: length = 336

2019-05-08 09:17:48 info capwap: application: CAPWAP: receive DCD send response to CAPWAP event!, length:66

2019-05-08 09:17:48 info capwap: application: receive event DCD send response to CAPWAP: eventid = 169: length = 66

2019-05-08 09:17:48 info capwap: application: CAPWAP: can not find the event request id (38).

2019-05-08 09:17:48 info capwap: application: CAPWAP: can not find the event query confirm information type request id (38).

2019-05-08 09:15:20 info ah_dcd: application: get track-ip trigger access console request: cancel.

2019-05-08 09:11:12 info last message repeated 3 times

2019-05-08 09:11:11 info kernel: [wifi]: [CLT_CAPS]Probe request from 48:f1:7f??e9:e6 is too short: 74

2019-05-08 09:10:20 info ah_dcd: application: get track-ip trigger access console request: cancel.

2019-05-08 09:10:12 info capwap: application: CAPWAP: receive capwap query interface map info response event!, length:1096

2019-05-08 09:10:12 info capwap: application: receive event capwap query interface map info response: eventid = 172: length = 1096

2019-05-08 09:09:03 info capwap: application: CAPWAP: receive RedSec get proxy info response event!, length:29

2019-05-08 09:09:03 info capwap: application: receive event RedSec get proxy info response: eventid = 240: length = 29

2019-05-08 09:09:03 info ah_scd: aaa: response to HM successfully, self isn't a elected Radsec proxy server.

2019-05-08 09:09:03 debug ah_scd: aaa: Get proxy server info success, 1 totally.

2019-05-08 09:09:02 info capwap: application: receive event statistics recv event to auth: eventid = 129: length = 48

2019-05-08 09:09:02 info capwap: application: CAPWAP receive IDP push all event!

2019-05-08 09:09:02 info capwap: application: receive event capwap_idp_pull_all: eventid = 122: length = 15

2019-05-08 09:09:02 info capwap: application: CAPWAP: receive DCD send response to CAPWAP event!, length:20

2019-05-08 09:09:02 info capwap: application: receive event DCD send response to CAPWAP: eventid = 169: length = 20

2019-05-08 09:09:02 info capwap: application: receive event statistics recv event to dcd: eventid = 125: length = 4

Re: Device AH-xxxxxx Connect State Change: The CAPWAP connection was lost.

samantha_lynn — Mon, 13 May 2019 21:55:39 GMT

Hello, thank you for that buffered log section. Mostly likely you are experiencing missed echo packets.

The HiveManager and the APs have a call and response check-in system that allows them to confirm that each side of the connection is still responsive; these call and response packets are called echos. There is a specific time window within which an AP would need to respond to an echo to be considered still responsive. If the AP misses a certain number of echos back to back, it is considered disconnected until it responds again. If an AP is still connected to the internet and passing traffic while missing echos, it's likely due to latency on the network; the AP is unable to process the echo request quickly enough due to the amount of traffic it has to process before responding. To confirm that we are missing echo packets, we'd want to SSH in to an AP to enable CAPWAP debugs and then collect tech data. 

CAPWAP debugs:

_debug capwap info

_debug capwap basic

_debug capwap stat

These will be erased if the AP reboots. You can also manually disable these at any time by adding "no" to the start of the command; for example "no _debug capwap info".

These guides review how to get tech data, depending on which HiveManager Platform you are using:

HiveManager Classic-https://thehivecommunity.aerohive.com/s/article/How-to-download-tech-data-in-Classic

HiveManager NG-https://thehivecommunity.aerohive.com/s/article/How-to-download-tech-data-in-HiveManager

Device CLI-https://thehivecommunity.aerohive.com/s/article/Collecting-Tech-Data-via-CLI

If you could send that tech data to me at communityhelp@aerohive.com, I can review it and let you know what we find.

Re: Device AH-xxxxxx Connect State Change: The CAPWAP connection was lost.

stewart_mahoney — Tue, 14 May 2019 15:35:35 GMT

Hi @Sam Pirok ,

Thank you for your reply.

I've been working with Ashley at Cloud and we think this could potentially be due to the SonicWall on site. 6/7 of my APs are now using HTTP rather than UDP 12222 and only 1/7 is using UDP and that is the only one causing me issues now. I am currently trying to get hold of the chap who manages our network to make the change to our firewall. If this fails, however, I will send you the requested data.

Thanks.

Re: Device AH-xxxxxx Connect State Change: The CAPWAP connection was lost.

tvilo — Wed, 15 May 2019 23:20:11 GMT

I have been having this same issue for almost a year now (with SonicWall on-site). Do you just change the transport method to http in the cli? Or is it an actual change to the firewall config?

Re: Device AH-xxxxxx Connect State Change: The CAPWAP connection was lost.

stewart_mahoney — Wed, 15 May 2019 23:30:57 GMT

Hi Tyler,

I seem to have resolved this now. The paragraph below is the one I followed which resolved it. What I think is happening is the SonicWall is changing the source port on some packets from the original UDP 12222 (can't tell you why it does that though) which is obviously causing issues.

"If persistent NAT is not enabled, the Sonicwall FW is allowed to change the NATted (source) port address, which intermittently breaks CAPWAP connection. There is a quick fix which is not obvious in the Sonicwall GUI. It’s under VoIP ->Settings -> General Settings -> “Enable consistent NAT”."

Once that's done, you should be able to change the transport method via the CLI and it not cause problems.

Give me a shout if that doesn't work.

Re: Device AH-xxxxxx Connect State Change: The CAPWAP connection was lost.

tvilo — Thu, 16 May 2019 00:24:32 GMT

Oh man you are definitely a lifesaver! I remember one time I had enabled Persistent NAT and I could have swore it caused other issues, but that was probably 7 SonicOS fw updates ago. Thank you again, I had been searching all over for a while and came across this one today.

Re: Device AH-xxxxxx Connect State Change: The CAPWAP connection was lost.

tvilo — Thu, 16 May 2019 00:34:35 GMT

It apparently tells you in the help menu. Last sentence of first paragraph.

With Consistent NAT enabled, all subsequent requests from either host 192.116.168.10 or 192.116.168.20 using the same ports illustrated in IP address and port pairs result in using the same translated address and port pairs. Without Consistent NAT, the port and possibly the IP address change with every request.

NOTE: Enabling Consistent NAT causes a slight decrease in overall security, because of the increased predictability of the address and port pairs. Most UDP-based applications are compatible with traditional NAT. Therefore, do not enable Consistent NAT unless your network uses applications that require it.