V2110 & AP3805i Poll Timeouts

[Fatson]

Greetings!

Have a V2110 virtual controller and a bunch ( ~ 300 ) 3805i APs.  Controller log is ful of  “Wireless AP session poll disconnect...”, so double checked for network issues, VM settings, POE & port configurations - had no evidences of smth. weird.  The issue regards to different APs, from different controllers on separate floors. Also, problem shows itself only on workhours, never at night or early morning.

But:

at the exact time of appearance of minor event  “Wireless AP session poll disconnect...” in controller logs, the traces from affected AP show the next lines:

Controller log:

06/01/21 16:40:00

Minor

RU Session Manager

Wireless AP session poll disconnect Wireless AP session timed out. (AP SN 17371500085N0000 AP NAME lp49-2fl-ap5 AP IP 10....68 timeout 45 secs (active tunnel))

---

AP Traces:

AP  LogLastReboot (UTC time, should add 3, so exact 16:39):

 Jun  1 13:39:17 krn: wmi_unified_cmd_send: MAX 1024 WMI Pending cmds reached.pending cmd 1023

…………

Jun  1 13:39:17 krn: QDF BUG in wmi_unified_cmd_send Line 1990

Jun  1 13:39:17 krn: Kernel bug detected[#1]:
 Jun  1 13:39:17 krn: Cpu 0
 Jun  1 13:39:17 krn: $ 0   : 00000000 00000000 00000030 00000001
 Jun  1 13:39:17 krn: $ 4   : 80242ad8 00000000 00000001 8010e00c
 Jun  1 13:39:17 krn: $ 8   : 0000000a 00000000 00000001 0000000c
 Jun  1 13:39:17 krn: $12   : 00000000 00000375 00000001 00000000
 Jun  1 13:39:17 krn: $16   : 8154c000 801d7cf0 8c690000 8c5c2034
 Jun  1 13:39:17 krn: $20   : 8c6f0000 00000000 8c6f4fa8 00000000
 Jun  1 13:39:17 krn: $24   : 00000000 8010d980                  
 Jun  1 13:39:17 krn: $28   : 8023c000 8023fcd0 ffffffff 8c6399b0
 Jun  1 13:39:17 krn: Hi    : 00000000
 Jun  1 13:39:17 krn: Lo    : b69f5000
 Jun  1 13:39:17 krn: epc   : 8c6399b0 wmi_unified_cmd_send+0x884/0x948 [qca_ol]
 Jun  1 13:39:17 krn:     Tainted: P          
 Jun  1 13:39:17 krn: ra    : 8c6399b0 wmi_unified_cmd_send+0x884/0x948 [qca_ol]
 Jun  1 13:39:17 krn: Status: 1100ff03    KERNEL EXL IE 
 Jun  1 13:39:17 krn: Cause : 00800024
 Jun  1 13:39:17 krn: PrId  : 00019750 (MIPS 74Kc)
 Jun  1 13:39:17 krn: Modules linked in: dpi_drv(P) chantry_ext(P) ath_pktlog(P) qca_da ath_dev(P) qca_ol ath_rate_atheros(P) umac hst_tx99(P) ath_dfs(P) ath_spectral(P) ath_hal(P) asf(P) qdf mem_manager(P) tunnelDrv(P) tlvDrv(P) ipsecXform(P) athrs_gma
 Jun  1 13:39:17 krn: Process swapper (pid: 0, threadinfo=8023c000, task=80240000, tls=00000000)

 Jun  1 13:39:17 krn: Stack : 3d201500 8c6904fc 000007c6 00000001 00000002 334556f9 000003ff 8023fd28
 Jun  1 13:39:17 krn:         8c350340 00000000 00000001 8023fd68 00200200 821b63c0 8c60d4c4 8c340340
 Jun  1 13:39:17 krn:         80210000 8c60c820 00000000 8023fd68 00000008 821b0000 00000001 00000000
 Jun  1 13:39:17 krn:         8c340340 00000100 80240000 8c60d5ac 00000375 821b5ba0 821b5a74 80240000
 Jun  1 13:39:17 krn:         821b5ba0 8002de34 00000000 821b5a70 00000375 24c6f173 8023fd68 8023fd68
 Jun  1 13:39:17 krn:         ...
 Jun  1 13:39:17 krn: Call Trace:
 Jun  1 13:39:17 krn: [<8c6399b0>] wmi_unified_cmd_send+0x884/0x948 [qca_ol]
 Jun  1 13:39:17 krn: [<8c60c820>] ol_ath_gpio_output+0x5c/0x74 [qca_ol]
 Jun  1 13:39:17 krn: [<8c60d5ac>] ol_ath_attach+0xd74/0xdec [qca_ol]
 Jun  1 13:39:17 krn: Code: 24841218  0220f809  240607c6 <0200000d> 0b18e66d  00000000  3c02801d  24841198  24427cf0 
 -----------------------------------------

AP report:

======== Access Point Problem Report =======

INFO: Linux version 2.6.31--10.41.05.0010 (root@ngap-fat6) (gcc version 4.9.2 (crosstool-NG 1.21.0) ) #2 Wed Mar 7 18:59:08 EST 2018
INFO: Last HW-WR 00000002->00004030 @ 0008ef90 from 8c738d10
INFO: Last HW-WR 00023f60->0000402c @ 0008ef90 from 8c738d6c
INFO: Last HW-WR 00023f60->00004034 @ 0008ef90 from 8c738d7c
INFO: Last HW-WR 00000002->0000403c @ 0008ef90 from 8c738d00
INFO: 17371500085N0000, AP3805i, jiffies 0008ef9b

Kernel bug detected[#1]:
Cpu 0
$ 0   : 00000000 00000000 00000030 00000001
$ 4   : 80242ad8 00000000 00000001 8010e00c
$ 8   : 0000000a 00000000 00000001 0000000c
$12   : 00000000 00000375 00000001 00000000
$16   : 8154c000 801d7cf0 8c690000 8c5c2034
$20   : 8c6f0000 00000000 8c6f4fa8 00000000
$24   : 00000000 8010d980                  
$28   : 8023c000 8023fcd0 ffffffff 8c6399b0
Hi    : 00000000
Lo    : b69f5000
epc   : 8c6399b0 wmi_unified_cmd_send+0x884/0x948 [qca_ol]
    Tainted: P          
ra    : 8c6399b0 wmi_unified_cmd_send+0x884/0x948 [qca_ol]
Status: 1100ff03    KERNEL EXL IE 
Cause : 00800024
PrId  : 00019750 (MIPS 74Kc)
Modules linked in: dpi_drv(P) chantry_ext(P) ath_pktlog(P) qca_da ath_dev(P) qca_ol ath_rate_atheros(P) umac hst_tx99(P) ath_dfs(P) ath_spectral(P) ath_hal(P) asf(P) qdf mem_manager(P) tunnelDrv(P) tlvDrv(P) ipsecXform(P) athrs_gmac ap_drv(P) libeCrypto(P) m2Task(P) atherosHooks(P) prodenv(P)
Process swapper (pid: 0, threadinfo=8023c000, task=80240000, tls=00000000)
Stack : 3d201500 8c6904fc 000007c6 00000001 00000002 334556f9 000003ff 8023fd28
        8c350340 00000000 00000001 8023fd68 00200200 821b63c0 8c60d4c4 8c340340
        80210000 8c60c820 00000000 8023fd68 00000008 821b0000 00000001 00000000
        8c340340 00000100 80240000 8c60d5ac 00000375 821b5ba0 821b5a74 80240000
        821b5ba0 8002de34 00000000 821b5a70 00000375 24c6f173 8023fd68 8023fd68
        ...
Call Trace:
[<8c6399b0>] wmi_unified_cmd_send+0x884/0x948 [qca_ol]
[<8c60c820>] ol_ath_gpio_output+0x5c/0x74 [qca_ol]
[<8c60d5ac>] ol_ath_attach+0xd74/0xdec [qca_ol]
---------------------------------------------------------------

I think this  kernel halt  drives AP to hang and then reboot, so it drops all clients, etc.

Can’t find any info about this thread “ wmi_unified_cmd_send” and what makes it crash the AP.

Again, this happens to various APs, this one is just a typical illustration.

Any advice, tip or help. Thanks in advance.

[Gareth_Mitchell]

Hi Alex

I’m glad that helped.  Generally speaking my recommendation would always be the latest release which is currently 10.51.18.

-Gareth

[Fatson]

Gareth, many, many thanks!!

Disable MFP did the trick and dready logs about Poll timeout disconnection are gone. APs aren’t rebooting anymore!

If you can, please suggest  me current soft/firmware for  V2110 virtual EWC and 3805 APs.

Big Regards, Alex

[Gareth_Mitchell · Accepted solution]

Hi

To get the best analysis GTAC would need to see the full AP log bundle, open a case and attach the logs there.

There are however a couple of things you could try:

1. Upgrade to the latest firmware. 
2. Disable MFP (management frame protection) on all wlan services.

-Gareth