topic Re: Howto debug AP disconnected from internet cloudiq in ExtremeCloud IQ

Howto debug AP disconnected from internet cloudiq

Leo_Bistmans1 — Tue, 18 Oct 2022 08:59:37 GMT

Since ExtremeNetworks forced us to use a remote Internet based cloudiq, often 10 or 15% of or AP show disconnected.

The internet cloudiq is only 7 ip hops away ( nl-gcp.extremecloudiq.com ), sub 10 milliseconds.

C:\WINDOWS>tracert -d nl-gcp.extremecloudiq.com

Tracing route to nl-gcp.extremecloudiq.com [34.91.82.64]
over a maximum of 30 hops:

1 <1 ms 3 ms 6 ms 10.12.255.253
2 3 ms 1 ms <1 ms 10.3.1.6
3 1 ms <1 ms <1 ms 172.31.44.3
4 2 ms 2 ms 1 ms 193.191.18.41
5 3 ms 13 ms 2 ms 10.28.35.54
6 24 ms 14 ms 6 ms 80.249.208.247
7 9 ms 9 ms 9 ms 34.91.82.64

Trace complete.

We can still get on the AP via SSH, and run commands like 'show capwap client'. Any hints welcome on how to trigger reconnect or test the connectivity from the AP command line.

Re: Howto debug AP disconnected from internet cloudiq

tandrews1 — Tue, 18 Oct 2022 12:39:02 GMT

Depending upon your particular network/firewall setup, investigate increasing the UDP session timeout.

A somewhat related thread that started us down that path:

https://extremeportal.force.com/ExtrArticleDetail?an=000058924

Re: Howto debug AP disconnected from internet cloudiq

jlmangas — Tue, 18 Oct 2022 12:55:24 GMT

Do you have more than one Internet connection which are balanced? If so, make sure the AP capwap connection goes out always using the same connection (meaning the same IP public address) otherwise the XIQ server might receive packets with different source addresses and think the AP is disconnected.

Re: Howto debug AP disconnected from internet cloudiq

Leo_Bistmans1 — Wed, 19 Oct 2022 14:20:37 GMT

Double checked, one outside/public IP. Even if we would change that, the uniqe AP id should make it irrelevant, right?

Re: Howto debug AP disconnected from internet cloudiq

Leo_Bistmans1 — Wed, 19 Oct 2022 14:22:37 GMT

Likely why smart people invented TCP 40 years ago 😉

Double checked the 300 second interval is there instead of the default 20 ( on Checkpoint firewall ).

Re: Howto debug AP disconnected from internet cloudiq

jlmangas — Thu, 20 Oct 2022 14:15:08 GMT

Does your ISP use CG-NAT? If so, the keepalive packets (UDP 12222) might arrive to XIQ cloud with different source port than the initial packets (That's because CG-NAT shares a public IP address among several users)

Re: Howto debug AP disconnected from internet cloudiq

Leo_Bistmans1 — Fri, 21 Oct 2022 08:20:00 GMT

No, we run the public ip address on our Checkpoint firewall.

I wonder if there is an api or other way to read the online/offline value out of the cloudiq on the internet, so I can plot values in a graph on Zabbix or such.

It would be nice to correlate that graph with other values like firewall cpu usage, latency / hop count to the cloudiq on the internet etc etc.

Re: Howto debug AP disconnected from internet cloudiq

Leo_Bistmans1 — Fri, 21 Oct 2022 12:49:47 GMT

Even the cloudiq on the internet is on different ip addresses ( in the same range )

I see 34.91.82.7, 72, 76 in the same second in a tcpdump capture.

Re: Howto debug AP disconnected from internet cloudiq

jlmangas — Mon, 24 Oct 2022 07:15:17 GMT

Hi,

ssh to one AP and enter the command:

capwap client transport HTTP

save config

This will use TCP port 80 for capwap connection instead of UDP 12222.

Check if you still see session disconnects.