Problems with old WLAN Clients after changing the WLAN infrastructure

[Anonymous]

Hello,

we switched our wlan infrastructure at one of our warehouses from old Motorola Symbol Aps using a WEP SSID to our new wlan infrastructure using the XIQ Cloud, with primarily AP410 and a PPSK SSID, using one PPSK for every forklift in the warehouse. (They use an old Win7 embedded OS).

After the switch we noticed several disconnects within the Logistics Software we are using on the terminals to manage the warehouse. We Software keeps losing the connection to the server for a few seconds (from 2-3 seconds to 15-20) and then reconnects. It seems like it’s happening all over the warehouse, even at locations where i can see with our other devices that the wlan connections seems good and stable. What i’ve noticed is that it looks like some sort of roaming problem to me. It seems to me like it is some sort of roaming problem. Maybe the terminals take too long to roam between the aps or having problems with PPSK authentication?

I checked the client monitor to check the clients and noticed a poor client health. Seeing many issues displayed as “The client is configured with an incorrect static IP address or gateway.” Which in fact, seems not to be correct. The forklifts are configured with a static ip, dns and gateway (we dont use DHCP in this VLAN). So what is the cause of this error? Could the be also a hint to our main wlan problem? 

If you need any more information, dont hesitate to ask. I appreciate every help. 🙂

[Ovais_Qayyum]

Ouch ….!! this is new 🙂

Yes, if you broadcast the same SSID on both bands and the clients are highly mobile, there is a good potential that you will see the clients jumping between the bands all the time. Mainly because you have different coverage overlaps for 2.4 and 5GHz bands (much smaller for 5GHz) and with Band Steering enabled it can make things difficult for roaming clients.    

If you can also get rid of SSIDs that are not used or can be consolidated by using “user profiles” in the configuration, it will greatly help you with the WLAN performance, because you have only three channels and a lot of APs causing too much CCI, having more SSIDs will increase mgmt. overhead traffic on the channel by quite a lot which means less airtime for clients. Based on the following rough overhead calculation, if you have 16 APs on the same channel (50APs/3 channels= 16 APs) with 4 SSIDs, the mgmt. overhead is almost 100%, which means almost no airtime for clients to send data. 

16

1 SSID= 37.84%

2 SSIDs= 75.69%

3 SSIDs=100.00%

4 SSIDs= 100.00%

In comparison to 2.4GHz, it's much better in 5GHz because you have a lot more channels to work with (Channels UNII-I 36,40,44,48 and UNII-III 149 to165). with the same 4 SSIDs the mgmt. overhead on the 5GHz network is only 26.69%.

16

1 SSID = 6.67%

2 SSID = 13.34%

3 SSIDs = 20.02%

4 SSIDs= 26.69%

If you think all clients can support 5GHz and 5GHz coverage is good, you should move this SSID to 5GHz. If not all the clients support 5GHz OR the 5GHz coverage is not uniform then only keep this SSID on 2.4GHz  (with 1 SSIDs and 16 APs sharing the channel the overhead is around 38%) and move the rest to 5GHz.

First, with the current PPSK SSID try broadcasting the SSID only in 2.4GHz or 5GHz (make sure you are using all non-DFS channels in 5GHz) based on what the clients support, monitor it, if it works, good deal. Otherwise, try the changes I mentioned in my previous post.

Regards,

Ovais

[Anonymous]

One more thing I’ve noticed yesterday. I saw some forklifts for a short time in 5 Ghz channels and so i checked the networks cards of the terminals again and noticed that they are indeed capable of 802.11a.

Could this also be causing some sort of trouble? Wifi1 is configured to 802.11ax

[Anonymous]

Hello Orvais,

i created the SSID Stapler with WEP to check if WPA2 was causing some of the problems we are facing. (Our old Infrastructure used WEP) and put only one of the forklifts in it. I guess this is also the reason why it the SSID consumes so much of the Airtime?

I was planing to change the SSID back to WPA2 PPSK anyway next week when i am on the plant. If i remember correctly, WPA2 and PPSK should make no difference in terms of authentication time, etc.? And i will also make every changes to the SSID like you suggested.

Here is the #show from the main AP in the loading hall:

#show int wifi0 _count
694331840 rx frames
28185661 rx data frames
        27797700 rx unicast data frames
        315217 rx multicast data frames
        72744 rx broadcast data frames
314482305 rx management frames
14698 rx control frames
0 rx unknown frames
14511 rx BAR (Block Ack Request) frames
694331840 11n rx frames
0 rx chip keyix errors
0 rx chip antenna errors
0 rx chip short frame errors
0 rx ieee queue depth
1363323 rx Retries
4% rx retry rate
103895730 rx CRC errors
13% rx CRC rate
49607 rx frame errors other than CRC
0 hw FIFO overrun
3568 decryption failed
0 MIC failure
194 rx frames dropped
        0 frame too short
        194 frame too large
        0 frame in other mode
0 rx dropped because no skbuff
40450324 tx data frames
        34967300 tx unicast data frames
                34967300 tx SU unicast data frames
                0 tx MU unicast data frames
        3955912 tx multicast data frames
        1527112 tx broadcast data frames
        --------------------------------
        40119242 tx WMM best effort data frames
                0 tx WMM best effort MU data frames
        54772 tx WMM background data frames
                0 tx WMM background MU data frames
        276310 tx WMM video data frames
                0 tx WMM video MU data frames
        0 tx WMM voice data frames
                0 tx WMM voice MU data frames
19961675 tx management frames other than beacon
166373135 tx beacon frames
0 tx beacon frame stucks
0 PCI errors
4504972 tx BAR (Block Ack Request) frames
21394174 tx aggregated completions
0 tx BAR (Block Ack Request) frames dropped
0 invalid tx rate index
0 invalid rx rate index
77947 tx retries
        56281 tx RTS failures
        21666 tx retries
0% tx retry rate
0% unicast data tx retry rate
0 tx retries of sub frames
6230461 tx frames with no ack marked
1544020 tx frames with rts enabled
0 tx frames with cts enabled
1428498 tx frames with short preamble
145945 tx frames with an alternate rate
0 tx frames with protection
34713 tx frames dropped
        3442 no tx buffer (data)
        31271 no tx buffer (mgmt)
4979728 tx frame errors
        4979728 too many hw retries
        0 hw FIFO underrun
        0 transmit filtered by hw
0 tx transmit queue full
1024 tx available buffers
0 exceeded txop
0 exceeded tx timer
10997867 tx frames succeed
5822143 tx frames failed
17 rx rssi from histogram
0 no skbuff available for beacon
0 switched from HT40 to HT20
0 switched from HT20 to HT40
1094489 channel switches
0 AMRP one-way neighbor timeout detected
1390 interference raise alert
1389 interference clear alert
0 band steering suppress
0 load balance suppress
347199 weak snr suppress
188161 safety net bypassed suppress
 

Another one within the halls:
#show int wifi0 _count
980976677 rx frames
3901107 rx data frames
        3761585 rx unicast data frames
        122019 rx multicast data frames
        17503 rx broadcast data frames
570644463 rx management frames
1877 rx control frames
0 rx unknown frames
1877 rx BAR (Block Ack Request) frames
980976677 11n rx frames
0 rx chip keyix errors
0 rx chip antenna errors
0 rx chip short frame errors
0 rx ieee queue depth
400458 rx Retries
10% rx retry rate
73190042 rx CRC errors
7% rx CRC rate
29554 rx frame errors other than CRC
0 hw FIFO overrun
1889 decryption failed
0 MIC failure
4 rx frames dropped
        0 frame too short
        4 frame too large
        0 frame in other mode
0 rx dropped because no skbuff
4142921 tx data frames
        2682117 tx unicast data frames
                2682117 tx SU unicast data frames
                0 tx MU unicast data frames
        984215 tx multicast data frames
        476589 tx broadcast data frames
        --------------------------------
        4046906 tx WMM best effort data frames
                0 tx WMM best effort MU data frames
        30958 tx WMM background data frames
                0 tx WMM background MU data frames
        65057 tx WMM video data frames
                0 tx WMM video MU data frames
        0 tx WMM voice data frames
                0 tx WMM voice MU data frames
9674441 tx management frames other than beacon
166265508 tx beacon frames
0 tx beacon frame stucks
0 PCI errors
1210803 tx BAR (Block Ack Request) frames
2209963 tx aggregated completions
0 tx BAR (Block Ack Request) frames dropped
0 invalid tx rate index
0 invalid rx rate index
33732 tx retries
        24795 tx RTS failures
        8937 tx retries
0% tx retry rate
1% unicast data tx retry rate
0 tx retries of sub frames
2401268 tx frames with no ack marked
604116 tx frames with rts enabled
0 tx frames with cts enabled
373706 tx frames with short preamble
69554 tx frames with an alternate rate
0 tx frames with protection
159 tx frames dropped
        0 no tx buffer (data)
        159 no tx buffer (mgmt)
1376147 tx frame errors
        1376147 too many hw retries
        0 hw FIFO underrun
        0 transmit filtered by hw
0 tx transmit queue full
1024 tx available buffers
0 exceeded txop
0 exceeded tx timer
3147329 tx frames succeed
9198734 tx frames failed
25 rx rssi from histogram
0 no skbuff available for beacon
0 switched from HT40 to HT20
0 switched from HT20 to HT40
1231140 channel switches
0 AMRP one-way neighbor timeout detected
533 interference raise alert
533 interference clear alert
0 band steering suppress
0 load balance suppress
411130 weak snr suppress
156358 safety net bypassed suppress
 

[Ovais_Qayyum]

It took me some time to check the packet capture, only one capture file had some meaningful data. The good thing is at least in the captured data the RF looks ok. I don’t see any retransmission of the frames and if you see the packet sequence, there is not much of a difference RSSI. it looks consistent.

One other thing that I have noticed is the channel utilization caused by the SSID “Stapler”, this SSID is comsuming most of the airtime on channel 11. I could only see channel 11 in the capture because the AP you might have used to capture the data was on channel 11 as well.   

The Block Ack that you mentioned in your previous response is not a concern, its just a mechanism for the AP to send acknowldements in blocks instead of individual acknowledgments which consumes more airtime. 

  I say you try following settings in the AP device template under radio interface settings:

Deny connection requests from legacy clients using - 11b (Enable to get rid of 11b legacy clients)

Background Scan - OFF

High Density Configuration - OFF

Band Steering - OFF

Client Load Balancing - OFF

Tx Beamforming - OFF

MU MIMO - OFF

Dynamic Channel Switching - OFF (keep it OFF for a while)

Weak Signal Probe Request Suppression - ON (set value to 15dbm first and then wait to see the diff., it should help free up the channels)

If all of this does not improve the situation then I would recommend to create a test WPA2 SSID and connect a test client with it to test the connectivity and see if faces the same problem.  

Send me output of #show int wifi0 counter from few APs where you see issues the most. 

Regards,

Ovais 

[Anonymous]

Hello Orvais,

if i remember correctly it was a some sort of redundancy coverage. Some areas had redundancy some not. But i am not shure and none of my colleagues is. 

One feedback i got yesterday from our plant is that in the loading street where we have a pretty empty hall, they also get many disconnects within the application. They also said the more forklifts in the loading hall the more disconnects. They even said that it looks like that only one forklift is able to work and then the next one “in the row” gets a connection and is able to work.

Most of the things i posted in here were from the “main” ap in the middle of the loading hall. It’s this area on the screenshot i posted, with three APs (channel 1,6 and 11). The ones you can see at the top or in the corners are either outdoor aps or in different halls.

I have checked the three aps in the hall right now and their summary state is on all three “good”. The CRC rate right now is between 8% and 17%. I will check these three aps again as soon as the number of forklifts in the loading hall rises up due to more trucks being loaded.

Could it help to configure everything down have the most sort of default wlan, atleast for the 2,4ghz? Like deactivating every sort of helping mechanism (short guard interval, band steering, etc.), activating every bitrate, raise the max. signal strength, since we only need the forklifts to be working properly.

Like i said, i am sure the old infrasctructure had massive overlaps in channels and cells. 

And no, sadly i dont know these antenna kits and couldn’t find them on the web so far. 

Can you check if this link works? I may have to creater longer captures:

https://www.dropbox.com/s/11vo1zmd8s3aie6/local-2021031575615UTC-BCF3103F6680-wifi0.pcap?dl=0

https://www.dropbox.com/s/ktl9mcvhgdz9dwp/local-20210312124219UTC-BCF3103F6680-wifi0.pcap?dl=0