Solved

Device AH-xxxxxx Connect State Change: The CAPWAP connection was lost.

  • 8 May 2019
  • 6 replies
  • 1218 views

Device AH-xxxxxx Connect State Change: The CAPWAP connection was lost.

 

I am getting the above error happen every 10-15 minutes on each of my connected APs. They all seem to work in that I can browse out onto the Internet, and have access to all relevant network resources when connected to them. All relevant traffic is allowed via our firewall.

 

Below are the logs I have found around the time it last happened:

 

2019-05-08 09:17:50 info  amrp2: l2routing: amrp receive capwap-statistic-request Mod_id (4) No.(183) ifindex 9

2019-05-08 09:17:49 info  capwap: application: CAPWAP: can not find the event query confirm information type request id (44).

2019-05-08 09:17:49 info  capwap: application: CAPWAP: can not find the event request id (39).

2019-05-08 09:17:49 info  capwap: application: CAPWAP: can not find the event query confirm information type request id (39).

2019-05-08 09:17:49 info  capwap: application: CAPWAP: can not find the event request id (40).

2019-05-08 09:17:49 info  capwap: application: CAPWAP: can not find the event query confirm information type request id (40).

2019-05-08 09:17:48 info  capwap: application: CAPWAP: receive DCD send response to CAPWAP event!, length:570

2019-05-08 09:17:48 info  capwap: application: receive event DCD send response to CAPWAP: eventid = 169: length = 570

2019-05-08 09:17:48 info  capwap: application: CAPWAP: receive DCD send response to CAPWAP event!, length:1308

2019-05-08 09:17:48 info  capwap: application: receive event DCD send response to CAPWAP: eventid = 169: length = 1308

2019-05-08 09:17:48 info  capwap: application: CAPWAP: receive DCD send response to CAPWAP event!, length:38

2019-05-08 09:17:48 info  capwap: application: receive event DCD send response to CAPWAP: eventid = 169: length = 38

2019-05-08 09:17:48 info  capwap: application: CAPWAP: receive DCD send response to CAPWAP event!, length:146

2019-05-08 09:17:48 info  capwap: application: receive event DCD send response to CAPWAP: eventid = 169: length = 146

2019-05-08 09:17:48 info  capwap: application: CAPWAP: receive DCD send response to CAPWAP event!, length:336

2019-05-08 09:17:48 info  capwap: application: receive event DCD send response to CAPWAP: eventid = 169: length = 336

2019-05-08 09:17:48 info  capwap: application: CAPWAP: receive DCD send response to CAPWAP event!, length:66

2019-05-08 09:17:48 info  capwap: application: receive event DCD send response to CAPWAP: eventid = 169: length = 66

2019-05-08 09:17:48 info  capwap: application: CAPWAP: can not find the event request id (38).

2019-05-08 09:17:48 info  capwap: application: CAPWAP: can not find the event query confirm information type request id (38).

2019-05-08 09:15:20 info  ah_dcd: application: get track-ip trigger access console request: cancel.

2019-05-08 09:11:12 info  last message repeated 3 times

2019-05-08 09:11:11 info  kernel: [wifi]: [CLT_CAPS]Probe request from 48:f1:7f🇨🇫e9:e6 is too short: 74

2019-05-08 09:10:20 info  ah_dcd: application: get track-ip trigger access console request: cancel.

2019-05-08 09:10:12 info  capwap: application: CAPWAP: receive capwap query interface map info response event!, length:1096

2019-05-08 09:10:12 info  capwap: application: receive event capwap query interface map info response: eventid = 172: length = 1096

2019-05-08 09:09:03 info  capwap: application: CAPWAP: receive RedSec get proxy info response event!, length:29

2019-05-08 09:09:03 info  capwap: application: receive event RedSec get proxy info response: eventid = 240: length = 29

2019-05-08 09:09:03 info  ah_scd: aaa: response to HM successfully, self isn't a elected Radsec proxy server.

2019-05-08 09:09:03 debug  ah_scd: aaa: Get proxy server info success, 1 totally.

2019-05-08 09:09:02 info  capwap: application: receive event statistics recv event to auth: eventid = 129: length = 48

2019-05-08 09:09:02 info  capwap: application: CAPWAP receive IDP push all event!

2019-05-08 09:09:02 info  capwap: application: receive event capwap_idp_pull_all: eventid = 122: length = 15

2019-05-08 09:09:02 info  capwap: application: CAPWAP: receive DCD send response to CAPWAP event!, length:20

2019-05-08 09:09:02 info  capwap: application: receive event DCD send response to CAPWAP: eventid = 169: length = 20

2019-05-08 09:09:02 info  capwap: application: receive event statistics recv event to dcd: eventid = 125: length = 4

icon

Best answer by stewart.mahoney 15 May 2019, 18:30

Hi Tyler,

 

I seem to have resolved this now. The paragraph below is the one I followed which resolved it. What I think is happening is the SonicWall is changing the source port on some packets from the original UDP 12222 (can't tell you why it does that though) which is obviously causing issues.

 

"If persistent NAT is not enabled, the Sonicwall FW is allowed to change the NATted (source) port address, which intermittently breaks CAPWAP connection. There is a quick fix which is not obvious in the Sonicwall GUI. It’s under VoIP ->Settings -> General Settings -> “Enable consistent NAT”."

 

Once that's done, you should be able to change the transport method via the CLI and it not cause problems.

 

Give me a shout if that doesn't work.

View original

6 replies

Userlevel 1

Hello, thank you for that buffered log section. Mostly likely you are experiencing missed echo packets.

 

The HiveManager and the APs have a call and response check-in system that allows them to confirm that each side of the connection is still responsive; these call and response packets are called echos. There is a specific time window within which an AP would need to respond to an echo to be considered still responsive. If the AP misses a certain number of echos back to back, it is considered disconnected until it responds again. If an AP is still connected to the internet and passing traffic while missing echos, it's likely due to latency on the network; the AP is unable to process the echo request quickly enough due to the amount of traffic it has to process before responding. To confirm that we are missing echo packets, we'd want to SSH in to an AP to enable CAPWAP debugs and then collect tech data.  

 

CAPWAP debugs: 

_debug capwap info 

_debug capwap basic 

_debug capwap stat 

 

These will be erased if the AP reboots. You can also manually disable these at any time by adding "no" to the start of the command; for example "no _debug capwap info".  

 

These guides review how to get tech data, depending on which HiveManager Platform you are using:

HiveManager Classic-https://thehivecommunity.aerohive.com/s/article/How-to-download-tech-data-in-Classic 

HiveManager NG-https://thehivecommunity.aerohive.com/s/article/How-to-download-tech-data-in-HiveManager 

Device CLI-https://thehivecommunity.aerohive.com/s/article/Collecting-Tech-Data-via-CLI

 

If you could send that tech data to me at communityhelp@aerohive.com, I can review it and let you know what we find.

Hi @Sam Pirok​ ,

 

Thank you for your reply.

 

I've been working with Ashley at Cloud and we think this could potentially be due to the SonicWall on site. 6/7 of my APs are now using HTTP rather than UDP 12222 and only 1/7 is using UDP and that is the only one causing me issues now. I am currently trying to get hold of the chap who manages our network to make the change to our firewall. If this fails, however, I will send you the requested data.

 

Thanks.

I have been having this same issue for almost a year now (with SonicWall on-site). Do you just change the transport method to http in the cli? Or is it an actual change to the firewall config?

Hi Tyler,

 

I seem to have resolved this now. The paragraph below is the one I followed which resolved it. What I think is happening is the SonicWall is changing the source port on some packets from the original UDP 12222 (can't tell you why it does that though) which is obviously causing issues.

 

"If persistent NAT is not enabled, the Sonicwall FW is allowed to change the NATted (source) port address, which intermittently breaks CAPWAP connection. There is a quick fix which is not obvious in the Sonicwall GUI. It’s under VoIP ->Settings -> General Settings -> “Enable consistent NAT”."

 

Once that's done, you should be able to change the transport method via the CLI and it not cause problems.

 

Give me a shout if that doesn't work.

Oh man you are definitely a lifesaver! I remember one time I had enabled Persistent NAT and I could have swore it caused other issues, but that was probably 7 SonicOS fw updates ago. Thank you again, I had been searching all over for a while and came across this one today.

It apparently tells you in the help menu. Last sentence of first paragraph.

 

With Consistent NAT enabled, all subsequent requests from either host 192.116.168.10 or 192.116.168.20 using the same ports illustrated in IP address and port pairs result in using the same translated address and port pairs. Without Consistent NAT, the port and possibly the IP address change with every request.

 

NOTE: Enabling Consistent NAT causes a slight decrease in overall security, because of the increased predictability of the address and port pairs. Most UDP-based applications are compatible with traditional NAT. Therefore, do not enable Consistent NAT unless your network uses applications that require it.

Reply