topic Re: ah_dcd: wifi: wifi0: Power 20 exceeds HW limit(reg 15, board 22), forced to 15 ah_top: CPU rate overload. ah_top: system: PM: the system cpu0 busy exceed 360 second. in Aerohive Migrated Content

ah_dcd: wifi: wifi0: Power 20 exceeds HW limit(reg 15, board 22), forced to 15 ah_top: CPU rate overload. ah_top: system: PM: the system cpu0 busy exceed 360 second.

Anonymous — Thu, 11 Apr 2019 15:29:05 GMT

After seting up a GRE Tunnel, i get ALOT of those messages like the ones in the Question, from every AP within the tunnel.

The APs are 330 or 250s. We are running Hivemanager Classic with the newest OS, so do the Aps (Upates two weekends ago)

What is the cause of this problem?

Re: ah_dcd: wifi: wifi0: Power 20 exceeds HW limit(reg 15, board 22), forced to 15 ah_top: CPU rate overload. ah_top: system: PM: the system cpu0 busy exceed 360 second.

samantha_lynn — Thu, 11 Apr 2019 17:40:11 GMT

The top message regarding your power 20 exceeding the HW limit, that is indicating that the power settings in the HiveManager are too high for the wireless standards associated with your country code (and therefore too high for local wireless standards) so it was adjusted down to the maximum allowed in your location.

The other two messages regarding your CPU are indicating you are experiencing CPU spikes. Some things to check/try for CPU spikes:

(Note: Disabling a function in the steps below is a temporary suggestion to identify the source of the problem, not a recommended permanent fix.)

If you are seeing any messages indicating CPU spike in regards to MDNDS services, try rebooting the AP. If the CPU gets high again quickly after the reboot, we might want to disable MDNDS traffic.
If you have WIPs enabled, try disabling it to see if that was the cause of the CPU spikes.
If you are using QoS in any user profiles, try removing that to see if that was the cause of the CPU spikes.
If you have AVS enabled, try disabling it to see if this is the cause of the CPU spikes.
If you're using band steering, try disabling this.
If you have multicast to unicast conversion enabled in the SSID settings, try disabling this.
Run the command "show system process state" on the CLI of the AP to see which processes are taking up the most CPU.
Run a remote sniffer packet capture to get an idea of the traffic we are passing and check for any particular clients flooding the network.

Re: ah_dcd: wifi: wifi0: Power 20 exceeds HW limit(reg 15, board 22), forced to 15 ah_top: CPU rate overload. ah_top: system: PM: the system cpu0 busy exceed 360 second.

Anonymous — Tue, 14 May 2019 20:29:58 GMT

Hello Sam,

sorry for waiting.

This is taken by a AP141, which are also having the stated problems.

The most of the utilization is taken by: Soft interrupt utilization

Process state:

Mem: 98504K used, 23400K free, 0K shrd, 3480K buff, 29632K cached

CPU: 2.9% usr 3.1% sys 0.0% nic 42.6% idle 0.0% io 0.0% irq 51.1% sirq

Load average: 1.95 1.98 2.56 2/236 5050

PID PPID USER STAT VSZ %MEM CPU %CPU COMMAND

3 2 root SW 0 0.0 0 18.0 [ksoftirqd/0]

1680 1 root S 30780 25.1 0 3.9 /sbin/mdnsd

1612 1578 root S 61836 50.5 0 3.7 /opt/ah/bin/amrp2

2290 1578 root S 53512 43.7 0 2.3 /opt/ah/bin/ah_nbr

1607 1578 root S 103m 87.0 0 1.5 /opt/ah/bin/ah_auth

1609 1578 root S 121m101.8 0 0.9 /opt/ah/bin/ah_dcd

5024 2755 root R 3656 2.9 0 0.9 top

1578 1 root S 73084 59.7 0 0.7 /opt/ah/bin/ah_top

2288 1578 root S 116m 97.2 0 0.5 /opt/ah/bin/capwap

1611 1578 root S 93064 76.1 0 0.5 /opt/ah/bin/ah_scd

1889 1 root S 82588 67.5 0 0.5 /opt/ah/bin/ah_vpn

1606 1578 root S 43724 35.7 0 0.5 /opt/ah/bin/ah_capture

1604 1578 root S 153m128.7 0 0.2 /opt/ah/bin/ah_brd

1603 1578 root S 54168 44.3 0 0.2 /opt/ah/bin/ah_dcm

2287 1578 admin S 92144 75.3 0 0.0 /opt/ah/bin/ah_webui

1613 1578 root S 69668 56.9 0 0.0 /opt/ah/bin/l7d

1882 1578 root S 62324 50.9 0 0.0 /opt/ah/bin/ah_ipfw

1608 1578 root S 62308 50.9 0 0.0 /opt/ah/bin/fed

1890 1 root S 61824 50.5 0 0.0 /opt/ah/bin/ah_ltr

2289 1578 root S 61788 50.5 0 0.0 /opt/ah/bin/capwap_srv

1891 1 root S 61156 50.0 0 0.0 /opt/ah/bin/ah_lcs

1888 1 root S 55992 45.7 0 0.0 /opt/ah/bin/ah_bgd

1610 1578 root S 44552 36.4 0 0.0 /opt/ah/bin/ah_ibeacon

1881 1578 root S 38796 31.7 0 0.0 /opt/ah/bin/dhcpc

1536 1 root S 36372 29.7 0 0.0 /opt/ah/bin/ah_monitor

2280 1578 root S 36092 29.5 0 0.0 /opt/ah/bin/ah_cli

2755 2754 root S 30144 24.6 0 0.0 -ah_cli_ui

1861 1 root S 30108 24.6 0 0.0 /opt/ah/bin/ntpclient

2712 1 root S 11524 9.4 0 0.0 /usr/local/bin/php-cgi -b 127.0.0.1:2008

2754 2750 admin S 10428 8.5 0 0.0 sshd: admin@pts/0

Already rebooted the AP. Nothing changed. sniffering a packet will me more difficult within our network. The other options are not set. The ap141 is on the version: HiveOS 6.5r12.221611

We are experiencing heavy capwap losses (5-10m) a few times a day.

Re: ah_dcd: wifi: wifi0: Power 20 exceeds HW limit(reg 15, board 22), forced to 15 ah_top: CPU rate overload. ah_top: system: PM: the system cpu0 busy exceed 360 second.

samantha_lynn — Tue, 14 May 2019 21:09:38 GMT

The heavy CAPWAP losses combined with your other issues sounds like latency on the backend network. We can confirm this is the case with your CAPWAP issues by enabling CAPWAP debugs, waiting for the issue to happen again, and pulling tech data. My best guess is that we're missing echo packets.

The HiveManager and the APs have a call and response check-in system that allows them to confirm that each side of the connection is still responsive; these call and response packets are called echos. There is a specific time window within which an AP would need to respond to an echo to be considered still responsive. If the AP misses a certain number of echos back to back, it is considered disconnected until it responds again. If an AP is still connected to the internet and passing traffic while missing echos, it's likely due to latency on the network; the AP is unable to process the echo request quickly enough due to the amount of traffic it has to process before responding.

To confirm all of this, please SSH in to the AP and run the following CAPWAP debugs:

CAPWAP debugs:

_debug capwap info

_debug capwap basic

_debug capwap stat

These will be erased if the AP reboots. You can also manually disable these at any time by adding "no" to the start of the command; for example "no _debug capwap info".

Then wait for the device to disconnect from the HiveManager and reconnect, then pull tech data. This guide reviews how to get tech data in HiveManager Classic- https://thehivecommunity.aerohive.com/s/article/How-to-download-tech-data-in-Classic

If you'd like to send the tech data to me directly, my email is communityhelp@aerohive.com.

Re: ah_dcd: wifi: wifi0: Power 20 exceeds HW limit(reg 15, board 22), forced to 15 ah_top: CPU rate overload. ah_top: system: PM: the system cpu0 busy exceed 360 second.

Anonymous — Tue, 14 May 2019 21:20:31 GMT

I have to add something. The AP250 are not experiencing this issue. They got a normal CPU rate (below 10%). They are on the same network as the other APs.

I will try the other suggestions later.

Re: ah_dcd: wifi: wifi0: Power 20 exceeds HW limit(reg 15, board 22), forced to 15 ah_top: CPU rate overload. ah_top: system: PM: the system cpu0 busy exceed 360 second.

Anonymous — Tue, 14 May 2019 21:24:12 GMT

EDIT:

I've found this log-message: kernel: [mesh]: set proxy : b4b6:76c4:c68d 4018:b1b2 :7d80 n/a flag 0x1402

What does this mean?

Re: ah_dcd: wifi: wifi0: Power 20 exceeds HW limit(reg 15, board 22), forced to 15 ah_top: CPU rate overload. ah_top: system: PM: the system cpu0 busy exceed 360 second.

samantha_lynn — Wed, 22 May 2019 00:35:15 GMT

That is indicating AMRP is setting a mesh proxy in the event mesh needs to be started.