Why are some WLANs not tunneling traffic on all access points?

  • 0
  • 2
  • Question
  • Updated 3 months ago
  • Answered

Hello everyone,

I have an RFS-4000 cluster managing a mix of 6532 and 6562 access points. I have 5 WLANs, all of which are enabled on all radios, and using the default profile for each access point type. All WLAN traffic is tunneled through the controller. The configuration is quite basic, and it has been up and running for several years, without any significant adjustments.

We had a UPS failure several months ago, and the primary controller, as well as about half of the access points lost power. After restoring power, we have two WLANs (out of five) that are acting funny. The main problem I am having is that sometimes, when clients are successfully associated on one of the affected WLANs, no traffic seems to be tunneled out to the rest of the network. DHCP fails immediately, and even when assigning a static IP address and DNS, the wireless client is unable to communicate with anything else on the network.

The thing that makes this particularly confusing is that other WLANs on the same access point function fine at the same time. And functionality on the affected WLAN can be rock solid when associating with a different access point. Given that the WLAN and AP policies are the same across the entire configuration, and all traffic is tunneled, I'm not understanding why the issue would only affect a subset of the WLANs on a subset of the APs.

My ability to perform trial and error troubleshooting is very limited, as I am not located at the site, and the facility operates 24/7. Therefore I'm trying to line up some specific ideas about thing I can investigate or try when I am able to schedule a maintenance window.

Has anyone else seen an issue like this before?  Any thoughts on a good way to start investigating?

Thank you,
Micah

Photo of Micah

Micah

  • 150 Points 100 badge 2x thumb
  • confused

Posted 3 months ago

  • 0
  • 2
Photo of Andrew Webster

Andrew Webster

  • 1,746 Points 1k badge 2x thumb
Sometimes the config isn't as basic as one might think.
Post the config for all to comment on.  The problem might be obvious.
Photo of Micah

Micah

  • 150 Points 100 badge 2x thumb

Hi Andrew,


Here's the config. I replaced some passwords with stars.


!### show running-config
!
! Configuration of RFS4000 version 5.4.4.0-007R
!
!
version 2.2
!
!
ip access-list BROADCAST-MULTICAST-CONTROL
 permit tcp any any rule-precedence 10 rule-description "permit all TCP traffic"
 permit udp any eq 67 any eq dhcpc rule-precedence 11 rule-description "permit DHCP replies"
 deny udp any range 137 138 any range 137 138 rule-precedence 20 rule-description "deny windows netbios"
 deny ip any 224.0.0.0/4 rule-precedence 21 rule-description "deny IP multicast"
 deny ip any host 255.255.255.255 rule-precedence 22 rule-description "deny IP local broadcast"
 permit ip any any rule-precedence 100 rule-description "permit all IP traffic"
!
mac access-list PERMIT-ARP-AND-IPv4
 permit any any type ip rule-precedence 10 rule-description "permit all IPv4 traffic"
 permit any any type arp rule-precedence 20 rule-description "permit all ARP traffic"
!
firewall-policy default
 no ip dos ipspoof
 no ip dos tcp-sequence-past-window
 no ip-mac conflict
 no firewall enable
 no stateful-packet-inspection-l2
!
!
mint-policy global-default
!
meshpoint-qos-policy default
!
wlan-qos-policy default
 qos trust dscp
 qos trust wmm
!
radio-qos-policy default
!
wlan AFUS-DMZ
 ssid AFUS-DMZ
 vlan 120
 bridging-mode tunnel
 encryption-type tkip
 authentication-type none
 wpa-wpa2 psk 0 ***************
!
wlan AFUS-GUEST
 ssid AFUS-GUEST
 vlan 100
 bridging-mode tunnel
 encryption-type tkip
 authentication-type none
 wpa-wpa2 psk 0 ***************
!
wlan AFUS-OFFICE
 ssid AFUS-OFFICE
 vlan 1
 bridging-mode tunnel
 encryption-type tkip-ccmp
 authentication-type none
 wpa-wpa2 psk 0 ***************
!
wlan AFUS-PROD
 ssid AFUS-PROD
 vlan 30
 bridging-mode tunnel
 encryption-type tkip-ccmp
 authentication-type none
 wpa-wpa2 psk 0 ***************
!
wlan AFUS-VOICE
 ssid AFUS-VOICE
 vlan 60
 bridging-mode tunnel
 encryption-type ccmp
 authentication-type none
 wpa-wpa2 psk 0 ***************
!
ap300 default-ap300
 interface radio1
 interface radio2
!
smart-rf-policy default
 sensitivity custom
 smart-ocs-monitoring frequency 2.4GHz 120
 smart-ocs-monitoring sample-count 2.4GHz 15
!
!
management-policy default
 no http server
 https server
 ssh
 user admin password 1 *************** role superuser access all
 no snmp-server manager v2
 snmp-server community 0 *************** ro
 snmp-server user snmptrap v3 encrypted des auth md5 0 motorola
 snmp-server user snmpmanager v3 encrypted des auth md5 0 motorola
!
l2tpv3 policy default
!
profile rfs4000 default-rfs4000
 ip name-server 10.200.196.1
 ip name-server 10.200.196.2
 ip domain-name agrana.net
 autoinstall configuration
 autoinstall firmware
 crypto ikev1 policy ikev1-default
  isakmp-proposal default encryption aes-256 group 2 hash sha
 crypto ikev2 policy ikev2-default
  isakmp-proposal default encryption aes-256 group 2 hash sha
 crypto ipsec transform-set default esp-aes-256 esp-sha-hmac
 crypto ikev1 remote-vpn
 crypto ikev2 remote-vpn
 crypto auto-ipsec-secure
 interface radio1
 interface radio2
 interface up1
  switchport mode access
  switchport access vlan 10
  ip dhcp trust
  qos trust dscp
  qos trust 802.1p
 interface ge1
  description "WLAN Trunk"
  switchport mode trunk
  switchport trunk native vlan 400
  no switchport trunk native tagged
  switchport trunk allowed vlan 1,30,60,100,120,400
  ip dhcp trust
  qos trust dscp
  qos trust 802.1p
 interface ge2
  description Management
  switchport mode access
  switchport access vlan 10
  ip dhcp trust
  qos trust dscp
  qos trust 802.1p
 interface ge3
  switchport mode access
  switchport access vlan 10
  ip dhcp trust
  qos trust dscp
  qos trust 802.1p
 interface ge4
  switchport mode access
  switchport access vlan 10
  ip dhcp trust
  qos trust dscp
  qos trust 802.1p
 interface ge5
  switchport mode access
  switchport access vlan 10
  ip dhcp trust
  qos trust dscp
  qos trust 802.1p
 interface wwan1
 interface pppoe1
 use firewall-policy default
 cluster name USLYWLAN
 cluster force-configured-state-delay 5
 logging on
 logging buffered debugging
 service pm sys-restart
 router ospf
!
profile ap81xx default-ap81xx
 autoinstall configuration
 autoinstall firmware
 crypto ikev1 policy ikev1-default
  isakmp-proposal default encryption aes-256 group 2 hash sha
 crypto ikev2 policy ikev2-default
  isakmp-proposal default encryption aes-256 group 2 hash sha
 crypto ipsec transform-set default esp-aes-256 esp-sha-hmac
 crypto ikev1 remote-vpn
 crypto ikev2 remote-vpn
 crypto auto-ipsec-secure
 interface radio1
 interface radio2
 interface radio3
 interface ge1
  ip dhcp trust
  qos trust dscp
  qos trust 802.1p
 interface ge2
  ip dhcp trust
  qos trust dscp
  qos trust 802.1p
 interface vlan1
  ip address dhcp
  ip address zeroconf secondary
  ip dhcp client request options all
 interface wwan1
 interface pppoe1
 use firewall-policy default
 logging on
 service pm sys-restart
 router ospf
!
profile ap71xx default-ap71xx
 autoinstall configuration
 autoinstall firmware
 crypto ikev1 policy ikev1-default
  isakmp-proposal default encryption aes-256 group 2 hash sha
 crypto ikev2 policy ikev2-default
  isakmp-proposal default encryption aes-256 group 2 hash sha
 crypto ipsec transform-set default esp-aes-256 esp-sha-hmac
 crypto ikev1 remote-vpn
 crypto ikev2 remote-vpn
 crypto auto-ipsec-secure
 interface radio1
 interface radio2
 interface radio3
 interface ge1
  ip dhcp trust
  qos trust dscp
  qos trust 802.1p
 interface ge2
  ip dhcp trust
  qos trust dscp
  qos trust 802.1p
 interface vlan1
  ip address dhcp
  ip address zeroconf secondary
  ip dhcp client request options all
 interface wwan1
 interface pppoe1
 use firewall-policy default
 logging on
 service pm sys-restart
 router ospf
!
profile ap6532 default-ap6532
 ip default-gateway 10.200.197.126
 autoinstall configuration
 autoinstall firmware
 crypto ikev1 policy ikev1-default
  isakmp-proposal default encryption aes-256 group 2 hash sha
 crypto ikev2 policy ikev2-default
  isakmp-proposal default encryption aes-256 group 2 hash sha
 crypto ipsec transform-set default esp-aes-256 esp-sha-hmac
 crypto ikev1 remote-vpn
 crypto ikev2 remote-vpn
 crypto auto-ipsec-secure
 crypto load-management
 interface radio1
  wlan AFUS-GUEST bss 1 primary
  wlan AFUS-OFFICE bss 2 primary
  wlan AFUS-PROD bss 3 primary
  wlan AFUS-VOICE bss 4 primary
  wlan AFUS-DMZ bss 5 primary
 interface radio2
  wlan AFUS-GUEST bss 1 primary
  wlan AFUS-OFFICE bss 2 primary
  wlan AFUS-PROD bss 3 primary
  wlan AFUS-VOICE bss 4 primary
  wlan AFUS-DMZ bss 5 primary
 interface ge1
  switchport mode access
  switchport access vlan 10
  ip dhcp trust
  qos trust dscp
  qos trust 802.1p
 interface vlan1
  ip address dhcp
  ip address zeroconf secondary
  ip dhcp client request options all
  shutdown
 interface vlan10
  description Management
 interface pppoe1
 use firewall-policy default
 logging on
 service pm sys-restart
 router ospf
!
profile ap650 default-ap650
 autoinstall configuration
 autoinstall firmware
 crypto ikev1 policy ikev1-default
  isakmp-proposal default encryption aes-256 group 2 hash sha
 crypto ikev2 policy ikev2-default
  isakmp-proposal default encryption aes-256 group 2 hash sha
 crypto ipsec transform-set default esp-aes-256 esp-sha-hmac
 crypto ikev1 remote-vpn
 crypto ikev2 remote-vpn
 crypto auto-ipsec-secure
 crypto load-management
 interface radio1
 interface radio2
 interface ge1
  ip dhcp trust
  qos trust dscp
  qos trust 802.1p
 interface vlan1
  ip address dhcp
  ip address zeroconf secondary
  ip dhcp client request options all
 interface pppoe1
 use firewall-policy default
 logging on
 service pm sys-restart
!
profile ap6521 default-ap6521
 autoinstall configuration
 autoinstall firmware
 interface radio1
 interface ge1
  ip dhcp trust
  qos trust dscp
  qos trust 802.1p
 interface vlan1
  ip address dhcp
  ip address zeroconf secondary
  ip dhcp client request options all
 interface pppoe1
 use firewall-policy default
 logging on
 service pm sys-restart
!
profile ap621 default-ap621
 autoinstall configuration
 autoinstall firmware
 interface radio1
 interface ge1
  ip dhcp trust
  qos trust dscp
  qos trust 802.1p
 interface vlan1
  ip address dhcp
  ip address zeroconf secondary
  ip dhcp client request options all
 use firewall-policy default
 logging on
 service pm sys-restart
!
profile ap6511 default-ap6511
 autoinstall configuration
 autoinstall firmware
 interface radio1
 interface up1
  ip dhcp trust
  qos trust dscp
  qos trust 802.1p
 interface fe1
  ip dhcp trust
  qos trust dscp
  qos trust 802.1p
 interface fe2
  ip dhcp trust
  qos trust dscp
  qos trust 802.1p
 interface fe3
  ip dhcp trust
  qos trust dscp
  qos trust 802.1p
 interface fe4
  ip dhcp trust
  qos trust dscp
  qos trust 802.1p
 interface vlan1
  ip address dhcp
  ip address zeroconf secondary
  ip dhcp client request options all
 interface pppoe1
 use firewall-policy default
 logging on
 service pm sys-restart
!
profile ap6562 default-ap6562
 ip default-gateway 10.200.197.126
 autoinstall configuration
 autoinstall firmware
 crypto ikev1 policy ikev1-default
  isakmp-proposal default encryption aes-256 group 2 hash sha
 crypto ikev2 policy ikev2-default
  isakmp-proposal default encryption aes-256 group 2 hash sha
 crypto ipsec transform-set default esp-aes-256 esp-sha-hmac
 crypto ikev1 remote-vpn
 crypto ikev2 remote-vpn
 crypto auto-ipsec-secure
 crypto load-management
 interface radio1
  placement outdoor
  wlan AFUS-GUEST bss 1 primary
  wlan AFUS-OFFICE bss 2 primary
  wlan AFUS-PROD bss 3 primary
  wlan AFUS-VOICE bss 4 primary
  wlan AFUS-DMZ bss 5 primary
 interface radio2
  placement outdoor
  wlan AFUS-GUEST bss 1 primary
  wlan AFUS-OFFICE bss 2 primary
  wlan AFUS-PROD bss 3 primary
  wlan AFUS-VOICE bss 4 primary
  wlan AFUS-DMZ bss 5 primary
 interface ge1
  switchport mode access
  switchport access vlan 10
  ip dhcp trust
  qos trust dscp
  qos trust 802.1p
 interface vlan1
  ip address dhcp
  ip address zeroconf secondary
  ip dhcp client request options all
  shutdown
 interface vlan10
  description Management
 interface pppoe1
 use firewall-policy default
 service pm sys-restart
!
profile ap6522 default-ap6522
 autoinstall configuration
 autoinstall firmware
 crypto ikev1 policy ikev1-default
  isakmp-proposal default encryption aes-256 group 2 hash sha
 crypto ikev2 policy ikev2-default
  isakmp-proposal default encryption aes-256 group 2 hash sha
 crypto ipsec transform-set default esp-aes-256 esp-sha-hmac
 crypto ikev1 remote-vpn
 crypto ikev2 remote-vpn
 crypto auto-ipsec-secure
 crypto load-management
 interface radio1
 interface radio2
 interface ge1
  ip dhcp trust
  qos trust dscp
  qos trust 802.1p
 interface vlan1
  ip address dhcp
  ip address zeroconf secondary
  ip dhcp client request options all
 interface pppoe1
 use firewall-policy default
 service pm sys-restart
 router ospf
!
profile ap622 default-ap622
 autoinstall configuration
 autoinstall firmware
 crypto ikev1 policy ikev1-default
  isakmp-proposal default encryption aes-256 group 2 hash sha
 crypto ikev2 policy ikev2-default
  isakmp-proposal default encryption aes-256 group 2 hash sha
 crypto ipsec transform-set default esp-aes-256 esp-sha-hmac
 crypto ikev1 remote-vpn
 crypto ikev2 remote-vpn
 crypto auto-ipsec-secure
 crypto load-management
 interface radio1
 interface radio2
 interface ge1
  ip dhcp trust
  qos trust dscp
  qos trust 802.1p
 interface vlan1
  ip address dhcp
  ip address zeroconf secondary
  ip dhcp client request options all
 use firewall-policy default
 logging on
 service pm sys-restart
!
rf-domain default
 location Lysander
 contact "Micah Clark"
 country-code us
 use smart-rf-policy default
 control-vlan 10
!
rfs4000 B4-C7-99-DD-49-EC
 use profile default-rfs4000
 use rf-domain default
 hostname USLYWLAN1
 license AP DEFAULT-6AP-LICENSE
 license AAP ************************
 license ADSEC DEFAULT-ADV-SEC-LICENSE
 ip default-gateway 10.200.197.126
 interface vlan10
  description Management
  ip address 10.200.197.51/25
  ip address zeroconf secondary
 cluster name USLYWLAN
 cluster mode active
 cluster member ip 10.200.197.51
 cluster member ip 10.200.197.52
 cluster member vlan 10
 cluster master-priority 250
 cluster handle-stp
 cluster force-configured-state-delay 5
 logging on
 logging console warnings
 logging buffered warnings
!
rfs4000 B4-C7-99-DD-4F-46
 use profile default-rfs4000
 use rf-domain default
 hostname USLYWLAN2
 license AP DEFAULT-6AP-LICENSE
 license ADSEC DEFAULT-ADV-SEC-LICENSE
 ip default-gateway 10.200.197.126
 interface vlan10
  ip address 10.200.197.52/25
  ip address zeroconf secondary
 cluster mode standby
 cluster member ip 10.200.197.51
 cluster member ip 10.200.197.52
 cluster member vlan 10
!
ap6532 84-24-8D-16-AF-94
 use profile default-ap6532
 use rf-domain default
 hostname USLYAP21
 area "Maintenance Office"
 interface radio1
  channel 11
  power 17
 interface radio2
 interface vlan10
  ip address 10.200.197.41/25
!
ap6532 B4-C7-99-9F-82-EC
 use profile default-ap6532
 use rf-domain default
 hostname USLYAP01
 area "Front Office - Cubes"
 interface radio1
  channel 11
  power 17
 interface vlan10
  ip address 10.200.197.21/25
 interface vlan100
  ip address 172.22.194.21/24
!
ap6532 B4-C7-99-9F-91-C8
 use profile default-ap6532
 use rf-domain default
 hostname USLYAP20
 area "Receiving Office"
 interface radio1
  channel 11
  power 2
 interface vlan10
  ip address 10.200.197.40/25
!
ap6532 B4-C7-99-A0-5D-60
 use profile default-ap6532
 use rf-domain default
 hostname USLYAP03
 area "Training Room"
 interface radio1
  channel 11
  power 8
 interface vlan10
  ip address 10.200.197.23/25
!
ap6532 B4-C7-99-A0-5D-68
 use profile default-ap6532
 use rf-domain default
 hostname USLYAP02
 area "Front Office - Break Rm."
 interface radio1
  channel 1
  power 17
 interface vlan10
  ip address 10.200.197.22/25
!
ap6562 FC-0A-81-17-28-8C
 use profile default-ap6562
 use rf-domain default
 hostname USLYAP04
 area "Flavor Room"
 interface radio1
  channel 6
  power 8
  placement outdoor
 interface radio2
  placement outdoor
 interface vlan10
  ip address 10.200.197.24/25
!
ap6562 FC-0A-81-17-29-5C
 use profile default-ap6562
 use rf-domain default
 hostname USLYAP09
 area "Tote Wash"
 interface radio1
  channel 11
  power 8
  placement outdoor
 interface radio2
  placement outdoor
 interface vlan10
  ip address 10.200.197.29/25
!
ap6562 FC-0A-81-17-29-94
 use profile default-ap6562
 use rf-domain default
 hostname USLYAP14
 area "Thaw Room"
 interface radio1
  channel 1
  power 8
  placement outdoor
 interface radio2
  placement outdoor
 interface vlan10
  ip address 10.200.197.34/25
!
ap6562 FC-0A-81-17-29-C4
 use profile default-ap6562
 use rf-domain default
 hostname USLYAP05
 area "Dry Storage"
 interface radio1
  channel 1
  power 8
  placement outdoor
 interface radio2
  placement outdoor
 interface vlan10
  ip address 10.200.197.25/25
!
ap6562 FC-0A-81-17-2A-3C
 use profile default-ap6562
 use rf-domain default
 hostname USLYAP16
 area "Cooler (SW)"
 interface radio1
  channel 6
  power 8
  placement outdoor
 interface radio2
  placement outdoor
 interface vlan10
  ip address 10.200.197.36/25
!
ap6562 FC-0A-81-17-2A-60
 use profile default-ap6562
 use rf-domain default
 hostname USLYAP13
 area "Processing (SE)"
 interface radio1
  channel 1
  power 8
  placement outdoor
 interface radio2
  placement outdoor
 interface vlan10
  ip address 10.200.197.33/25
!
ap6562 FC-0A-81-17-3E-34
 use profile default-ap6562
 use rf-domain default
 hostname USLYAP12
 area Freezer
 interface radio1
  channel 6
  power 8
  placement outdoor
 interface radio2
  placement outdoor
 interface vlan10
  ip address 10.200.197.32/25
!
ap6562 FC-0A-81-17-48-1C
 use profile default-ap6562
 use rf-domain default
 hostname USLYAP11
 area "Processing (NW)"
 interface radio1
  channel 6
  power 8
  placement outdoor
 interface radio2
  placement outdoor
 interface vlan10
  ip address 10.200.197.31/25
!
ap6562 FC-0A-81-17-48-B8
 use profile default-ap6562
 use rf-domain default
 hostname USLYAP19
 area SPARE
 interface radio1
  placement outdoor
 interface radio2
  placement outdoor
 interface vlan10
  ip address 10.200.197.39/25
!
ap6562 FC-0A-81-17-49-A0
 use profile default-ap6562
 use rf-domain default
 hostname USLYAP10
 area Allergen
 interface radio1
  channel 11
  power 2
  placement outdoor
 interface radio2
  placement outdoor
 interface vlan10
  ip address 10.200.197.30/25
!
ap6562 FC-0A-81-17-4A-A0
 use profile default-ap6562
 use rf-domain default
 hostname USLYAP07
 area Unitizing
 interface radio1
  channel 6
  power 8
  placement outdoor
 interface radio2
  placement outdoor
 interface vlan10
  ip address 10.200.197.27/25
!
ap6562 FC-0A-81-17-4A-A8
 use profile default-ap6562
 use rf-domain default
 hostname USLYAP15
 area "Cooler (NE)"
 interface radio1
  channel 11
  power 8
  placement outdoor
 interface radio2
  placement outdoor
 interface vlan10
  ip address 10.200.197.35/25
!
ap6562 FC-0A-81-17-7D-74
 use profile default-ap6562
 use rf-domain default
 hostname USLYAP08
 area Shipping
 interface radio1
  channel 1
  power 8
  placement outdoor
 interface radio2
  placement outdoor
 interface vlan10
  ip address 10.200.197.28/25
!
ap6562 FC-0A-81-17-7E-04
 use profile default-ap6562
 use rf-domain default
 hostname USLYAP17
 area "Ext. Tote Storage"
 interface radio1
  channel 1
  power 8
  placement outdoor
 interface radio2
  placement outdoor
 interface vlan10
  ip address 10.200.197.37/25
!
ap6562 FC-0A-81-17-7E-F8
 use profile default-ap6562
 use rf-domain default
 hostname USLYAP06
 area "Dry Receiving"
 interface radio1
  shutdown
  channel 6
  power 8
  placement outdoor
  antenna-gain 0.0
 interface radio2
  placement outdoor
 interface vlan10
  ip address 10.200.197.26/25
!
ap6562 FC-0A-81-17-97-28
 use profile default-ap6562
 use rf-domain default
 hostname USLYAP18
 area "Tote Receiving"
 interface radio1
  channel 1
  power 30
  placement outdoor
 interface radio2
  placement outdoor
 interface vlan10
  ip address 10.200.197.38/25
!
!
end

Photo of Andrew Webster

Andrew Webster

  • 1,746 Points 1k badge 2x thumb
Hi Micah,
The first thing I would check are the network switches into which the RFS4000s are connected to be sure that the VLAN settings on the ports weren't lost because of the power outage.

After that, there are a number of potential "issues" with the config; here are some things to think about...
I noticed that the configuration in the RFS4000 profile seems to indicate that the RFS4000s are possibly connected into the network with more than one port.  This type of connection should be avoided as it can cause ports to be shutdown unexpectedly because of spanning-tree.
A better configuration is to only use the UP1 port and have it configured with all the VLANs, including AP adoption VLAN.  In your case it appears as if UP1 is only for AP adoption and GE1 is for everything else.  

There are some unusual power settings on some of the APs, some are at minimum power, others at maximum, and one is even shut down.  Check and review power settings to ensure that the RF signal is covering the space adequately.  Consider having an "exit survey" done by a wireless professional.

I noticed that radio 2 (5GHz) isn't configured anywhere, meaning it will use smart-rf to auto channel/power.  If you don't physically have 5GHz antennas connected to the AP6562, or use 5GHz wireless clients, consider shutting down the radio, as dual-band clients could see the 5GHz radio when very close by, but won't get any usability out of it.

Other troubleshooting tips
Is the cluster working properly?  Check output of "show cluster members".  
If the cluster becomes broken (or one of the RFSes is no longer in cluster), the license pool only lasts 100 days, after which they revert to their native license quantities.  Check output of "show licenses".
Check output of "show adoption status" to ensure that all the expected APs are showing up.
(Edited)
Photo of Micah

Micah

  • 150 Points 100 badge 2x thumb

Thank you so much for all of the suggestions, Andrew!!

Some responses and additional questions:

1. I checked the switch port configurations, and they are as they should be. Identical for both controllers, and matching the controller configuration.

2. I will look for an opportunity to move the adoption VLAN onto the same port with everything else, as a best practice. I don't think spanning tree has been hurting me so far though, so I have not seen an indication in the logs of spanning tree adjustments or of ports going up/down while experiencing these issues on the network.

3. The unusual power settings are the result of a relatively long process of tweaking things to fix coverage issues in and around the building. There are a lot of concrete and metal walls with large doors that open and close, which leads to a constantly varying footprint for many of the access points. It took a lot of tuning to optimize the footprint of each AP, but we finally got it to a good place where client connectivity was stable, which was quite a while before this problem started. We did have a wireless professional help us with that, using real-time heat map measurements.

4. The issues we had that required extensive tuning only seemed to be impacting our 2.4GHz clients. We do have 5GHz clients, but we never seemed to have any connectivity issues with them, so I never messed with those radio settings. I think this is perhaps because the 5GHz band has more non-overlapping channels, so the smart-rf is better able to handle that automatically?

5. From what you say about cluster licensing, I think this may be the root of our problem. Right now the secondary controller is offline. I thought these units acted in a simple active/standby manner, and had no idea licensing would be affected when one of them is offline. We had a packet storm several months ago, and shutting down the switch ports to the standby controller resolved the issue. I haven't had an opportunity for a maintenance window to reboot it and bring it back online since then, so it has just been sitting offline for several months, possibly more than 100 days. I have a maintenance window this Sunday, so I can bring it back online then. Do you know if the cluster will automatically sort out its licensing status when the second unit comes back online, or is there something I need to do to get them synced up again on licensing?

Here's the output regarding cluster and licensing:

USLYWLAN1#show cluster configuration

Cluster Configuration Information
 Name                         : USLYWLAN
 Configured Mode              : Active
 Master Priority              : 250
 Force configured state       : Disabled
 Force configured state delay : 5 minutes
 Handle STP                   : Enabled
USLYWLAN1#show cluster status

Cluster Runtime Information
 Protocol version             : 1
 Cluster operational state    : active
 AP license                   : 12
 AAP license                  : 12
 AP count                     : 0
 AAP count                    : 19
 Max AP adoption capacity     : 36
 Number of connected member(s): 0
USLYWLAN1#show cluster members
------------------------------------------------------------------------------------------
   HOSTNAME     MEMBER-ID            MAC           MASTER  OPERATIONAL-STATE   LAST-SEEN
------------------------------------------------------------------------------------------
  USLYWLAN1    19.DD.49.EC    B4-C7-99-DD-49-EC    True     active             self
  USLYWLAN2                   B4-C7-99-DD-4F-46    False    down
------------------------------------------------------------------------------------------
USLYWLAN1#show licenses
Serial Number : 13158522400016

Device Licenses:
  AP-LICENSE
    String     : DEFAULT-6AP-LICENSE
    Value      : 6
  AAP-LICENSE
    String     : ***************************
    Value      : 12
  ADVANCED-SECURITY
    String     : DEFAULT-ADV-SEC-LICENSE

Cluster Licenses:
  AP-LICENSE
    Value      : 12
    Used       : 7
  AAP-LICENSE
    Value      : 12
    Used       : 12


Active Members:
--------------------------------------------------------------------------------
        MEMBER            SERIAL       AP LIC   AAP LIC   NO.APS     NO.AAPS
--------------------------------------------------------------------------------
  B4-C7-99-DD-49-EC   13158522400016   6        12        0        19
--------------------------------------------------------------------------------

Non-Active Members:
--------------------------------------------------------------------------------
        MEMBER              SERIAL        AP LIC   AAP LIC     VALIDITY(HRS)
--------------------------------------------------------------------------------
  B4-C7-99-DD-4F-46    13179522400028    6         0          1

--------------------------------------------------------------------------------
USLYWLAN1#


Thank you!!
Micah

Photo of Micah

Micah

  • 150 Points 100 badge 2x thumb

Also, here is the current adoption status.


USLYWLAN1#show adoption status
----------------------------------------------------------------------------------------------------------
AP-NAME           VERSION         CFG-STAT         ADOPTED-BY        LAST-ADOPTION                  UPTIME
----------------------------------------------------------------------------------------------------------
USLYAP21          5.4.4.0-007R    configured       USLYWLAN1         2017-11-09 17:43:17  97 days 20:48:42
USLYAP01          5.4.4.0-007R    configured       USLYWLAN1         2017-11-02 07:05:26 105 days 08:35:51
USLYAP20          5.4.4.0-007R    configured       USLYWLAN1         2017-11-02 07:05:30 113 days 19:54:07
USLYAP02          5.4.4.0-007R    configured       USLYWLAN1         2017-11-02 07:05:29 105 days 08:35:49
USLYAP04          5.4.4.0-007R    configured       USLYWLAN1         2017-11-06 22:08:12 100 days 16:23:11
USLYAP09          5.4.4.0-007R    configured       USLYWLAN1         2017-11-02 07:05:25 105 days 08:35:53
USLYAP14          5.4.4.0-007R    configured       USLYWLAN1         2017-11-02 07:05:31 910 days 01:23:09
USLYAP05          5.4.4.0-007R    configured       USLYWLAN1         2017-11-02 07:05:30 105 days 08:35:52
USLYAP16          5.4.4.0-007R    configured       USLYWLAN1         2017-11-02 07:05:32 910 days 01:35:15
USLYAP13          5.4.4.0-007R    configured       USLYWLAN1         2017-11-02 07:05:26 910 days 01:26:17
USLYAP12          5.4.4.0-007R    configured       USLYWLAN1         2017-11-02 07:05:36 910 days 01:28:08
USLYAP11          5.4.4.0-007R    configured       USLYWLAN1         2017-11-02 07:05:36 910 days 01:24:08
USLYAP10          5.4.4.0-007R    configured       USLYWLAN1         2017-11-02 07:05:25 105 days 08:35:53
USLYAP07          5.4.4.0-007R    configured       USLYWLAN1         2017-11-02 07:05:22 105 days 08:35:42
USLYAP15          5.4.4.0-007R    configured       USLYWLAN1         2017-11-02 07:05:26 910 days 01:29:38
USLYAP08          5.4.4.0-007R    configured       USLYWLAN1         2017-11-02 07:05:30 910 days 01:31:17
USLYAP17          5.4.4.0-007R    configured       USLYWLAN1         2017-11-02 07:05:22 113 days 20:00:59
USLYAP06          5.4.4.0-007R    configured       USLYWLAN1         2017-11-02 07:05:21 105 days 08:35:53
USLYAP18          5.4.4.0-007R    configured       USLYWLAN1         2017-11-02 07:05:21 113 days 19:39:13
------------------------------------------------------------------------------------------------------------
Total number of APs displayed: 19
USLYWLAN1#

Photo of Andrew Webster

Andrew Webster

  • 1,746 Points 1k badge 2x thumb
Hi Micah,

It looks as if you have a total of 21 APs in your config, but only 19 APs are adopted.
Use: "show wireless ap configured" and "show adoption offline" to see which ones are missing.
On the primary RFS, You have a license for 18 APs (default 6 + 12 additional), and the default 6 APs on the secondary RFS, making a total of 24 when everything is working properly.
However, the license pool (validity period) appears to be expiring in 1 hour (so it has been 100 days), in which case you will not be able to continue to adopt all 19 APs, and one AP will be dropped, so you are about to experience more problems.  

The fact that you mention that you had disabled switch ports because of a broadcast storm problem points back to the fact that the RFSes are connected in duplicate into the network...not ideal, but you'll have to live with it until you can change the topology.   I'd also be very careful of the 'cluster handle-stp' that is present in your configuration.
One suggestion would be to restart the second RFS, enable the network ports, if only for a minute or two, just to re-sync the license pool until you can deal with the issue in a more permanent fashion.
You could also look at your switches' spanning tree status on the ports facing the RFSes to ensure none are in blocking or alternate.

I have seen issues with older versions of RFS4000 code (5.4.x specifically) where two cluster members would get into a shouting match with each other and send >10,000 packets/sec to the broadcast address, thus creating what appears to be a broadcast storm.  
I would suggest you upgrade to a more recent firmware version, in part to stabilise the cluster and additionally to address the WPA2 KRACK vulnerability (your auditors will be happy... see: https://extremeportal.force.com/ExtrArticleDetail?n=000018005)
The upgrade should be seamless, but use the RFS4000 LEAN image and  load AP firmware for AP6532 and AP6562 into the RFSes once they have been upgraded so that the APs can also be upgraded.
Photo of Micah

Micah

  • 150 Points 100 badge 2x thumb


Andrew,

Thank you so much for your observations!

The two access points that were already offline were known (one is a spare). You pointed out the 18/19 issue just in time though, and I disabled one additional access point that is not needed right now, so I should be safe at 18 for the moment.

During my maintenance window this weekend, I should be able to bring the secondary online to re-sync the license status, and to adjust the ports on both controllers to use just one trunk interface. I will also schedule a time to do a code upgrade, but that will probably have to wait a while longer. The shouting match scenario you mentioned seems plausible. I believe we have seen that issue twice over 4 years. The first time a reboot resolved it.

I see the 'cluster handle-stp' line in the config, but it's not familiar to me. I am seeing some references to it online, but nothing explaining what it does. Would it be better if I disabled that, and just let the switches handle stp, especially after the controllers are only on a single interface each?


Many thanks,
Micah

Photo of Doug Hyde

Doug Hyde, Technical Support Manager

  • 20,056 Points 20k badge 2x thumb
Hello Micah, 

Were you able to resolve this issue? 
Photo of Micah

Micah

  • 150 Points 100 badge 2x thumb
Hi Doug, thank you for following up. I was able to repair the cluster over the weekend, but it doesn't seem to have helped with the main problem. My next step will be to upgrade the code to the latest version. It will probably take me several weeks to get another downtime window scheduled though.
Photo of Andrew Webster

Andrew Webster

  • 1,746 Points 1k badge 2x thumb
Extreme has just written an article specifically dealing with licenses across clusters.
https://extremeportal.force.com/ExtrArticleDetail?n=000021938