V2110 & AP3805i Poll Timeouts

Fatson — Tue, 01 Jun 2021 22:15:03 GMT

Greetings!

Have a V2110 virtual controller and a bunch ( ~ 300 ) 3805i APs. Controller log is ful of “Wireless AP session poll disconnect...”, so double checked for network issues, VM settings, POE & port configurations - had no evidences of smth. weird. The issue regards to different APs, from different controllers on separate floors. Also, problem shows itself only on workhours, never at night or early morning.

But:

at the exact time of appearance of minor event “Wireless AP session poll disconnect...” in controller logs, the traces from affected AP show the next lines:

Controller log:

06/01/21 16:40:00

Minor

RU Session Manager

Wireless AP session poll disconnect Wireless AP session timed out. (AP SN 17371500085N0000 AP NAME lp49-2fl-ap5 AP IP 10....68 timeout 45 secs (active tunnel))

AP Traces:

AP LogLastReboot (UTC time, should add 3, so exact 16:39):

Jun 1 13:39:17 krn: wmi_unified_cmd_send: MAX 1024 WMI Pending cmds reached.pending cmd 1023

…………

Jun 1 13:39:17 krn: QDF BUG in wmi_unified_cmd_send Line 1990

Jun 1 13:39:17 krn: Kernel bug detected[#1]:
Jun 1 13:39:17 krn: Cpu 0
Jun 1 13:39:17 krn: $ 0 : 00000000 00000000 00000030 00000001
Jun 1 13:39:17 krn: $ 4 : 80242ad8 00000000 00000001 8010e00c
Jun 1 13:39:17 krn: $ 8 : 0000000a 00000000 00000001 0000000c
Jun 1 13:39:17 krn: $12 : 00000000 00000375 00000001 00000000
Jun 1 13:39:17 krn: $16 : 8154c000 801d7cf0 8c690000 8c5c2034
Jun 1 13:39:17 krn: $20 : 8c6f0000 00000000 8c6f4fa8 00000000
Jun 1 13:39:17 krn: $24 : 00000000 8010d980
Jun 1 13:39:17 krn: $28 : 8023c000 8023fcd0 ffffffff 8c6399b0
Jun 1 13:39:17 krn: Hi : 00000000
Jun 1 13:39:17 krn: Lo : b69f5000
Jun 1 13:39:17 krn: epc : 8c6399b0 wmi_unified_cmd_send+0x884/0x948 [qca_ol]
Jun 1 13:39:17 krn: Tainted: P
Jun 1 13:39:17 krn: ra : 8c6399b0 wmi_unified_cmd_send+0x884/0x948 [qca_ol]
Jun 1 13:39:17 krn: Status: 1100ff03 KERNEL EXL IE
Jun 1 13:39:17 krn: Cause : 00800024
Jun 1 13:39:17 krn: PrId : 00019750 (MIPS 74Kc)
Jun 1 13:39:17 krn: Modules linked in: dpi_drv(P) chantry_ext(P) ath_pktlog(P) qca_da ath_dev(P) qca_ol ath_rate_atheros(P) umac hst_tx99(P) ath_dfs(P) ath_spectral(P) ath_hal(P) asf(P) qdf mem_manager(P) tunnelDrv(P) tlvDrv(P) ipsecXform(P) athrs_gma
Jun 1 13:39:17 krn: Process swapper (pid: 0, threadinfo=8023c000, task=80240000, tls=00000000)

Jun 1 13:39:17 krn: Stack : 3d201500 8c6904fc 000007c6 00000001 00000002 334556f9 000003ff 8023fd28
Jun 1 13:39:17 krn: 8c350340 00000000 00000001 8023fd68 00200200 821b63c0 8c60d4c4 8c340340
Jun 1 13:39:17 krn: 80210000 8c60c820 00000000 8023fd68 00000008 821b0000 00000001 00000000
Jun 1 13:39:17 krn: 8c340340 00000100 80240000 8c60d5ac 00000375 821b5ba0 821b5a74 80240000
Jun 1 13:39:17 krn: 821b5ba0 8002de34 00000000 821b5a70 00000375 24c6f173 8023fd68 8023fd68
Jun 1 13:39:17 krn: ...
Jun 1 13:39:17 krn: Call Trace:
Jun 1 13:39:17 krn: [<8c6399b0>] wmi_unified_cmd_send+0x884/0x948 [qca_ol]
Jun 1 13:39:17 krn: [<8c60c820>] ol_ath_gpio_output+0x5c/0x74 [qca_ol]
Jun 1 13:39:17 krn: [<8c60d5ac>] ol_ath_attach+0xd74/0xdec [qca_ol]
Jun 1 13:39:17 krn: Code: 24841218 0220f809 240607c6 <0200000d> 0b18e66d 00000000 3c02801d 24841198 24427cf0
-----------------------------------------

AP report:

======== Access Point Problem Report =======

INFO: Linux version 2.6.31--10.41.05.0010 (root@ngap-fat6) (gcc version 4.9.2 (crosstool-NG 1.21.0) ) #2 Wed Mar 7 18:59:08 EST 2018
INFO: Last HW-WR 00000002->00004030 @ 0008ef90 from 8c738d10
INFO: Last HW-WR 00023f60->0000402c @ 0008ef90 from 8c738d6c
INFO: Last HW-WR 00023f60->00004034 @ 0008ef90 from 8c738d7c
INFO: Last HW-WR 00000002->0000403c @ 0008ef90 from 8c738d00
INFO: 17371500085N0000, AP3805i, jiffies 0008ef9b

Kernel bug detected[#1]:
Cpu 0
$ 0 : 00000000 00000000 00000030 00000001
$ 4 : 80242ad8 00000000 00000001 8010e00c
$ 8 : 0000000a 00000000 00000001 0000000c
$12 : 00000000 00000375 00000001 00000000
$16 : 8154c000 801d7cf0 8c690000 8c5c2034
$20 : 8c6f0000 00000000 8c6f4fa8 00000000
$24 : 00000000 8010d980
$28 : 8023c000 8023fcd0 ffffffff 8c6399b0
Hi : 00000000
Lo : b69f5000
epc : 8c6399b0 wmi_unified_cmd_send+0x884/0x948 [qca_ol]
Tainted: P
ra : 8c6399b0 wmi_unified_cmd_send+0x884/0x948 [qca_ol]
Status: 1100ff03 KERNEL EXL IE
Cause : 00800024
PrId : 00019750 (MIPS 74Kc)
Modules linked in: dpi_drv(P) chantry_ext(P) ath_pktlog(P) qca_da ath_dev(P) qca_ol ath_rate_atheros(P) umac hst_tx99(P) ath_dfs(P) ath_spectral(P) ath_hal(P) asf(P) qdf mem_manager(P) tunnelDrv(P) tlvDrv(P) ipsecXform(P) athrs_gmac ap_drv(P) libeCrypto(P) m2Task(P) atherosHooks(P) prodenv(P)
Process swapper (pid: 0, threadinfo=8023c000, task=80240000, tls=00000000)
Stack : 3d201500 8c6904fc 000007c6 00000001 00000002 334556f9 000003ff 8023fd28
8c350340 00000000 00000001 8023fd68 00200200 821b63c0 8c60d4c4 8c340340
80210000 8c60c820 00000000 8023fd68 00000008 821b0000 00000001 00000000
8c340340 00000100 80240000 8c60d5ac 00000375 821b5ba0 821b5a74 80240000
821b5ba0 8002de34 00000000 821b5a70 00000375 24c6f173 8023fd68 8023fd68
...
Call Trace:
[<8c6399b0>] wmi_unified_cmd_send+0x884/0x948 [qca_ol]
[<8c60c820>] ol_ath_gpio_output+0x5c/0x74 [qca_ol]
[<8c60d5ac>] ol_ath_attach+0xd74/0xdec [qca_ol]
---------------------------------------------------------------

I think this kernel halt drives AP to hang and then reboot, so it drops all clients, etc.

Can’t find any info about this thread “ wmi_unified_cmd_send” and what makes it crash the AP.

Again, this happens to various APs, this one is just a typical illustration.

Any advice, tip or help. Thanks in advance.

Re: V2110 & AP3805i Poll Timeouts

Gareth_Mitchell — Wed, 02 Jun 2021 23:52:54 GMT

To get the best analysis GTAC would need to see the full AP log bundle, open a case and attach the logs there.

There are however a couple of things you could try:

Upgrade to the latest firmware.
Disable MFP (management frame protection) on all wlan services.

-Gareth

Re: V2110 & AP3805i Poll Timeouts

Fatson — Thu, 03 Jun 2021 21:38:11 GMT

Gareth, many, many thanks!!

Disable MFP did the trick and dready logs about Poll timeout disconnection are gone. APs aren’t rebooting anymore!

If you can, please suggest me current soft/firmware for V2110 virtual EWC and 3805 APs.

Big Regards, Alex

Re: V2110 & AP3805i Poll Timeouts

Gareth_Mitchell — Fri, 04 Jun 2021 15:42:41 GMT

Hi Alex

I’m glad that helped. Generally speaking my recommendation would always be the latest release which is currently 10.51.18.

-Gareth

topic Re: V2110 & AP3805i Poll Timeouts in ExtremeWireless (General)

V2110 & AP3805i Poll Timeouts

Re: V2110 & AP3805i Poll Timeouts

Re: V2110 & AP3805i Poll Timeouts

Re: V2110 & AP3805i Poll Timeouts