topic RE: Why are some WLANs not tunneling traffic on all access points? in ExtremeWireless (WiNG)

Why are some WLANs not tunneling traffic on all access points?

Micah — Thu, 15 Feb 2018 02:17:00 GMT

Hello everyone,

I have an RFS-4000 cluster managing a mix of 6532 and 6562 access points. I have 5 WLANs, all of which are enabled on all radios, and using the default profile for each access point type. All WLAN traffic is tunneled through the controller. The configuration is quite basic, and it has been up and running for several years, without any significant adjustments.

We had a UPS failure several months ago, and the primary controller, as well as about half of the access points lost power. After restoring power, we have two WLANs (out of five) that are acting funny. The main problem I am having is that sometimes, when clients are successfully associated on one of the affected WLANs, no traffic seems to be tunneled out to the rest of the network. DHCP fails immediately, and even when assigning a static IP address and DNS, the wireless client is unable to communicate with anything else on the network.

The thing that makes this particularly confusing is that other WLANs on the same access point function fine at the same time. And functionality on the affected WLAN can be rock solid when associating with a different access point. Given that the WLAN and AP policies are the same across the entire configuration, and all traffic is tunneled, I'm not understanding why the issue would only affect a subset of the WLANs on a subset of the APs.

My ability to perform trial and error troubleshooting is very limited, as I am not located at the site, and the facility operates 24/7. Therefore I'm trying to line up some specific ideas about thing I can investigate or try when I am able to schedule a maintenance window.

Has anyone else seen an issue like this before? Any thoughts on a good way to start investigating?

Thank you,
Micah

RE: Why are some WLANs not tunneling traffic on all access points?

Andrew_Webster — Thu, 15 Feb 2018 03:11:00 GMT

Sometimes the config isn't as basic as one might think.
Post the config for all to comment on. The problem might be obvious.

RE: Why are some WLANs not tunneling traffic on all access points?

Micah — Thu, 15 Feb 2018 06:29:00 GMT

Hi Andrew,

Here's the config. I replaced some passwords with stars.

!### show running-config
!
! Configuration of RFS4000 version 5.4.4.0-007R
!
!
version 2.2
!
!
ip access-list BROADCAST-MULTICAST-CONTROL
permit tcp any any rule-precedence 10 rule-description "permit all TCP traffic"
permit udp any eq 67 any eq dhcpc rule-precedence 11 rule-description "permit DHCP replies"
deny udp any range 137 138 any range 137 138 rule-precedence 20 rule-description "deny windows netbios"
deny ip any 224.0.0.0/4 rule-precedence 21 rule-description "deny IP multicast"
deny ip any host 255.255.255.255 rule-precedence 22 rule-description "deny IP local broadcast"
permit ip any any rule-precedence 100 rule-description "permit all IP traffic"
!
mac access-list PERMIT-ARP-AND-IPv4
permit any any type ip rule-precedence 10 rule-description "permit all IPv4 traffic"
permit any any type arp rule-precedence 20 rule-description "permit all ARP traffic"
!
firewall-policy default
no ip dos ipspoof
no ip dos tcp-sequence-past-window
no ip-mac conflict
no firewall enable
no stateful-packet-inspection-l2
!
!
mint-policy global-default
!
meshpoint-qos-policy default
!
wlan-qos-policy default
qos trust dscp
qos trust wmm
!
radio-qos-policy default
!
wlan AFUS-DMZ
ssid AFUS-DMZ
vlan 120
bridging-mode tunnel
encryption-type tkip
authentication-type none
wpa-wpa2 psk 0 ***************
!
wlan AFUS-GUEST
ssid AFUS-GUEST
vlan 100
bridging-mode tunnel
encryption-type tkip
authentication-type none
wpa-wpa2 psk 0 ***************
!
wlan AFUS-OFFICE
ssid AFUS-OFFICE
vlan 1
bridging-mode tunnel
encryption-type tkip-ccmp
authentication-type none
wpa-wpa2 psk 0 ***************
!
wlan AFUS-PROD
ssid AFUS-PROD
vlan 30
bridging-mode tunnel
encryption-type tkip-ccmp
authentication-type none
wpa-wpa2 psk 0 ***************
!
wlan AFUS-VOICE
ssid AFUS-VOICE
vlan 60
bridging-mode tunnel
encryption-type ccmp
authentication-type none
wpa-wpa2 psk 0 ***************
!
ap300 default-ap300
interface radio1
interface radio2
!
smart-rf-policy default
sensitivity custom
smart-ocs-monitoring frequency 2.4GHz 120
smart-ocs-monitoring sample-count 2.4GHz 15
!
!
management-policy default
no http server
https server
ssh
user admin password 1 *************** role superuser access all
no snmp-server manager v2
snmp-server community 0 *************** ro
snmp-server user snmptrap v3 encrypted des auth md5 0 motorola
snmp-server user snmpmanager v3 encrypted des auth md5 0 motorola
!
l2tpv3 policy default
!
profile rfs4000 default-rfs4000
ip name-server 10.200.196.1
ip name-server 10.200.196.2
ip domain-name agrana.net
autoinstall configuration
autoinstall firmware
crypto ikev1 policy ikev1-default
isakmp-proposal default encryption aes-256 group 2 hash sha
crypto ikev2 policy ikev2-default
isakmp-proposal default encryption aes-256 group 2 hash sha
crypto ipsec transform-set default esp-aes-256 esp-sha-hmac
crypto ikev1 remote-vpn
crypto ikev2 remote-vpn
crypto auto-ipsec-secure
interface radio1
interface radio2
interface up1
switchport mode access
switchport access vlan 10
ip dhcp trust
qos trust dscp
qos trust 802.1p
interface ge1
description "WLAN Trunk"
switchport mode trunk
switchport trunk native vlan 400
no switchport trunk native tagged
switchport trunk allowed vlan 1,30,60,100,120,400
ip dhcp trust
qos trust dscp
qos trust 802.1p
interface ge2
description Management
switchport mode access
switchport access vlan 10
ip dhcp trust
qos trust dscp
qos trust 802.1p
interface ge3
switchport mode access
switchport access vlan 10
ip dhcp trust
qos trust dscp
qos trust 802.1p
interface ge4
switchport mode access
switchport access vlan 10
ip dhcp trust
qos trust dscp
qos trust 802.1p
interface ge5
switchport mode access
switchport access vlan 10
ip dhcp trust
qos trust dscp
qos trust 802.1p
interface wwan1
interface pppoe1
use firewall-policy default
cluster name USLYWLAN
cluster force-configured-state-delay 5
logging on
logging buffered debugging
service pm sys-restart
router ospf
!
profile ap81xx default-ap81xx
autoinstall configuration
autoinstall firmware
crypto ikev1 policy ikev1-default
isakmp-proposal default encryption aes-256 group 2 hash sha
crypto ikev2 policy ikev2-default
isakmp-proposal default encryption aes-256 group 2 hash sha
crypto ipsec transform-set default esp-aes-256 esp-sha-hmac
crypto ikev1 remote-vpn
crypto ikev2 remote-vpn
crypto auto-ipsec-secure
interface radio1
interface radio2
interface radio3
interface ge1
ip dhcp trust
qos trust dscp
qos trust 802.1p
interface ge2
ip dhcp trust
qos trust dscp
qos trust 802.1p
interface vlan1
ip address dhcp
ip address zeroconf secondary
ip dhcp client request options all
interface wwan1
interface pppoe1
use firewall-policy default
logging on
service pm sys-restart
router ospf
!
profile ap71xx default-ap71xx
autoinstall configuration
autoinstall firmware
crypto ikev1 policy ikev1-default
isakmp-proposal default encryption aes-256 group 2 hash sha
crypto ikev2 policy ikev2-default
isakmp-proposal default encryption aes-256 group 2 hash sha
crypto ipsec transform-set default esp-aes-256 esp-sha-hmac
crypto ikev1 remote-vpn
crypto ikev2 remote-vpn
crypto auto-ipsec-secure
interface radio1
interface radio2
interface radio3
interface ge1
ip dhcp trust
qos trust dscp
qos trust 802.1p
interface ge2
ip dhcp trust
qos trust dscp
qos trust 802.1p
interface vlan1
ip address dhcp
ip address zeroconf secondary
ip dhcp client request options all
interface wwan1
interface pppoe1
use firewall-policy default
logging on
service pm sys-restart
router ospf
!
profile ap6532 default-ap6532
ip default-gateway 10.200.197.126
autoinstall configuration
autoinstall firmware
crypto ikev1 policy ikev1-default
isakmp-proposal default encryption aes-256 group 2 hash sha
crypto ikev2 policy ikev2-default
isakmp-proposal default encryption aes-256 group 2 hash sha
crypto ipsec transform-set default esp-aes-256 esp-sha-hmac
crypto ikev1 remote-vpn
crypto ikev2 remote-vpn
crypto auto-ipsec-secure
crypto load-management
interface radio1
wlan AFUS-GUEST bss 1 primary
wlan AFUS-OFFICE bss 2 primary
wlan AFUS-PROD bss 3 primary
wlan AFUS-VOICE bss 4 primary
wlan AFUS-DMZ bss 5 primary
interface radio2
wlan AFUS-GUEST bss 1 primary
wlan AFUS-OFFICE bss 2 primary
wlan AFUS-PROD bss 3 primary
wlan AFUS-VOICE bss 4 primary
wlan AFUS-DMZ bss 5 primary
interface ge1
switchport mode access
switchport access vlan 10
ip dhcp trust
qos trust dscp
qos trust 802.1p
interface vlan1
ip address dhcp
ip address zeroconf secondary
ip dhcp client request options all
shutdown
interface vlan10
description Management
interface pppoe1
use firewall-policy default
logging on
service pm sys-restart
router ospf
!
profile ap650 default-ap650
autoinstall configuration
autoinstall firmware
crypto ikev1 policy ikev1-default
isakmp-proposal default encryption aes-256 group 2 hash sha
crypto ikev2 policy ikev2-default
isakmp-proposal default encryption aes-256 group 2 hash sha
crypto ipsec transform-set default esp-aes-256 esp-sha-hmac
crypto ikev1 remote-vpn
crypto ikev2 remote-vpn
crypto auto-ipsec-secure
crypto load-management
interface radio1
interface radio2
interface ge1
ip dhcp trust
qos trust dscp
qos trust 802.1p
interface vlan1
ip address dhcp
ip address zeroconf secondary
ip dhcp client request options all
interface pppoe1
use firewall-policy default
logging on
service pm sys-restart
!
profile ap6521 default-ap6521
autoinstall configuration
autoinstall firmware
interface radio1
interface ge1
ip dhcp trust
qos trust dscp
qos trust 802.1p
interface vlan1
ip address dhcp
ip address zeroconf secondary
ip dhcp client request options all
interface pppoe1
use firewall-policy default
logging on
service pm sys-restart
!
profile ap621 default-ap621
autoinstall configuration
autoinstall firmware
interface radio1
interface ge1
ip dhcp trust
qos trust dscp
qos trust 802.1p
interface vlan1
ip address dhcp
ip address zeroconf secondary
ip dhcp client request options all
use firewall-policy default
logging on
service pm sys-restart
!
profile ap6511 default-ap6511
autoinstall configuration
autoinstall firmware
interface radio1
interface up1
ip dhcp trust
qos trust dscp
qos trust 802.1p
interface fe1
ip dhcp trust
qos trust dscp
qos trust 802.1p
interface fe2
ip dhcp trust
qos trust dscp
qos trust 802.1p
interface fe3
ip dhcp trust
qos trust dscp
qos trust 802.1p
interface fe4
ip dhcp trust
qos trust dscp
qos trust 802.1p
interface vlan1
ip address dhcp
ip address zeroconf secondary
ip dhcp client request options all
interface pppoe1
use firewall-policy default
logging on
service pm sys-restart
!
profile ap6562 default-ap6562
ip default-gateway 10.200.197.126
autoinstall configuration
autoinstall firmware
crypto ikev1 policy ikev1-default
isakmp-proposal default encryption aes-256 group 2 hash sha
crypto ikev2 policy ikev2-default
isakmp-proposal default encryption aes-256 group 2 hash sha
crypto ipsec transform-set default esp-aes-256 esp-sha-hmac
crypto ikev1 remote-vpn
crypto ikev2 remote-vpn
crypto auto-ipsec-secure
crypto load-management
interface radio1
placement outdoor
wlan AFUS-GUEST bss 1 primary
wlan AFUS-OFFICE bss 2 primary
wlan AFUS-PROD bss 3 primary
wlan AFUS-VOICE bss 4 primary
wlan AFUS-DMZ bss 5 primary
interface radio2
placement outdoor
wlan AFUS-GUEST bss 1 primary
wlan AFUS-OFFICE bss 2 primary
wlan AFUS-PROD bss 3 primary
wlan AFUS-VOICE bss 4 primary
wlan AFUS-DMZ bss 5 primary
interface ge1
switchport mode access
switchport access vlan 10
ip dhcp trust
qos trust dscp
qos trust 802.1p
interface vlan1
ip address dhcp
ip address zeroconf secondary
ip dhcp client request options all
shutdown
interface vlan10
description Management
interface pppoe1
use firewall-policy default
service pm sys-restart
!
profile ap6522 default-ap6522
autoinstall configuration
autoinstall firmware
crypto ikev1 policy ikev1-default
isakmp-proposal default encryption aes-256 group 2 hash sha
crypto ikev2 policy ikev2-default
isakmp-proposal default encryption aes-256 group 2 hash sha
crypto ipsec transform-set default esp-aes-256 esp-sha-hmac
crypto ikev1 remote-vpn
crypto ikev2 remote-vpn
crypto auto-ipsec-secure
crypto load-management
interface radio1
interface radio2
interface ge1
ip dhcp trust
qos trust dscp
qos trust 802.1p
interface vlan1
ip address dhcp
ip address zeroconf secondary
ip dhcp client request options all
interface pppoe1
use firewall-policy default
service pm sys-restart
router ospf
!
profile ap622 default-ap622
autoinstall configuration
autoinstall firmware
crypto ikev1 policy ikev1-default
isakmp-proposal default encryption aes-256 group 2 hash sha
crypto ikev2 policy ikev2-default
isakmp-proposal default encryption aes-256 group 2 hash sha
crypto ipsec transform-set default esp-aes-256 esp-sha-hmac
crypto ikev1 remote-vpn
crypto ikev2 remote-vpn
crypto auto-ipsec-secure
crypto load-management
interface radio1
interface radio2
interface ge1
ip dhcp trust
qos trust dscp
qos trust 802.1p
interface vlan1
ip address dhcp
ip address zeroconf secondary
ip dhcp client request options all
use firewall-policy default
logging on
service pm sys-restart
!
rf-domain default
location Lysander
contact "Micah Clark"
country-code us
use smart-rf-policy default
control-vlan 10
!
rfs4000 B4-C7-99-DD-49-EC
use profile default-rfs4000
use rf-domain default
hostname USLYWLAN1
license AP DEFAULT-6AP-LICENSE
license AAP ************************
license ADSEC DEFAULT-ADV-SEC-LICENSE
ip default-gateway 10.200.197.126
interface vlan10
description Management
ip address 10.200.197.51/25
ip address zeroconf secondary
cluster name USLYWLAN
cluster mode active
cluster member ip 10.200.197.51
cluster member ip 10.200.197.52
cluster member vlan 10
cluster master-priority 250
cluster handle-stp
cluster force-configured-state-delay 5
logging on
logging console warnings
logging buffered warnings
!
rfs4000 B4-C7-99-DD-4F-46
use profile default-rfs4000
use rf-domain default
hostname USLYWLAN2
license AP DEFAULT-6AP-LICENSE
license ADSEC DEFAULT-ADV-SEC-LICENSE
ip default-gateway 10.200.197.126
interface vlan10
ip address 10.200.197.52/25
ip address zeroconf secondary
cluster mode standby
cluster member ip 10.200.197.51
cluster member ip 10.200.197.52
cluster member vlan 10
!
ap6532 84-24-8D-16-AF-94
use profile default-ap6532
use rf-domain default
hostname USLYAP21
area "Maintenance Office"
interface radio1
channel 11
power 17
interface radio2
interface vlan10
ip address 10.200.197.41/25
!
ap6532 B4-C7-99-9F-82-EC
use profile default-ap6532
use rf-domain default
hostname USLYAP01
area "Front Office - Cubes"
interface radio1
channel 11
power 17
interface vlan10
ip address 10.200.197.21/25
interface vlan100
ip address 172.22.194.21/24
!
ap6532 B4-C7-99-9F-91-C8
use profile default-ap6532
use rf-domain default
hostname USLYAP20
area "Receiving Office"
interface radio1
channel 11
power 2
interface vlan10
ip address 10.200.197.40/25
!
ap6532 B4-C7-99-A0-5D-60
use profile default-ap6532
use rf-domain default
hostname USLYAP03
area "Training Room"
interface radio1
channel 11
power 8
interface vlan10
ip address 10.200.197.23/25
!
ap6532 B4-C7-99-A0-5D-68
use profile default-ap6532
use rf-domain default
hostname USLYAP02
area "Front Office - Break Rm."
interface radio1
channel 1
power 17
interface vlan10
ip address 10.200.197.22/25
!
ap6562 FC-0A-81-17-28-8C
use profile default-ap6562
use rf-domain default
hostname USLYAP04
area "Flavor Room"
interface radio1
channel 6
power 8
placement outdoor
interface radio2
placement outdoor
interface vlan10
ip address 10.200.197.24/25
!
ap6562 FC-0A-81-17-29-5C
use profile default-ap6562
use rf-domain default
hostname USLYAP09
area "Tote Wash"
interface radio1
channel 11
power 8
placement outdoor
interface radio2
placement outdoor
interface vlan10
ip address 10.200.197.29/25
!
ap6562 FC-0A-81-17-29-94
use profile default-ap6562
use rf-domain default
hostname USLYAP14
area "Thaw Room"
interface radio1
channel 1
power 8
placement outdoor
interface radio2
placement outdoor
interface vlan10
ip address 10.200.197.34/25
!
ap6562 FC-0A-81-17-29-C4
use profile default-ap6562
use rf-domain default
hostname USLYAP05
area "Dry Storage"
interface radio1
channel 1
power 8
placement outdoor
interface radio2
placement outdoor
interface vlan10
ip address 10.200.197.25/25
!
ap6562 FC-0A-81-17-2A-3C
use profile default-ap6562
use rf-domain default
hostname USLYAP16
area "Cooler (SW)"
interface radio1
channel 6
power 8
placement outdoor
interface radio2
placement outdoor
interface vlan10
ip address 10.200.197.36/25
!
ap6562 FC-0A-81-17-2A-60
use profile default-ap6562
use rf-domain default
hostname USLYAP13
area "Processing (SE)"
interface radio1
channel 1
power 8
placement outdoor
interface radio2
placement outdoor
interface vlan10
ip address 10.200.197.33/25
!
ap6562 FC-0A-81-17-3E-34
use profile default-ap6562
use rf-domain default
hostname USLYAP12
area Freezer
interface radio1
channel 6
power 8
placement outdoor
interface radio2
placement outdoor
interface vlan10
ip address 10.200.197.32/25
!
ap6562 FC-0A-81-17-48-1C
use profile default-ap6562
use rf-domain default
hostname USLYAP11
area "Processing (NW)"
interface radio1
channel 6
power 8
placement outdoor
interface radio2
placement outdoor
interface vlan10
ip address 10.200.197.31/25
!
ap6562 FC-0A-81-17-48-B8
use profile default-ap6562
use rf-domain default
hostname USLYAP19
area SPARE
interface radio1
placement outdoor
interface radio2
placement outdoor
interface vlan10
ip address 10.200.197.39/25
!
ap6562 FC-0A-81-17-49-A0
use profile default-ap6562
use rf-domain default
hostname USLYAP10
area Allergen
interface radio1
channel 11
power 2
placement outdoor
interface radio2
placement outdoor
interface vlan10
ip address 10.200.197.30/25
!
ap6562 FC-0A-81-17-4A-A0
use profile default-ap6562
use rf-domain default
hostname USLYAP07
area Unitizing
interface radio1
channel 6
power 8
placement outdoor
interface radio2
placement outdoor
interface vlan10
ip address 10.200.197.27/25
!
ap6562 FC-0A-81-17-4A-A8
use profile default-ap6562
use rf-domain default
hostname USLYAP15
area "Cooler (NE)"
interface radio1
channel 11
power 8
placement outdoor
interface radio2
placement outdoor
interface vlan10
ip address 10.200.197.35/25
!
ap6562 FC-0A-81-17-7D-74
use profile default-ap6562
use rf-domain default
hostname USLYAP08
area Shipping
interface radio1
channel 1
power 8
placement outdoor
interface radio2
placement outdoor
interface vlan10
ip address 10.200.197.28/25
!
ap6562 FC-0A-81-17-7E-04
use profile default-ap6562
use rf-domain default
hostname USLYAP17
area "Ext. Tote Storage"
interface radio1
channel 1
power 8
placement outdoor
interface radio2
placement outdoor
interface vlan10
ip address 10.200.197.37/25
!
ap6562 FC-0A-81-17-7E-F8
use profile default-ap6562
use rf-domain default
hostname USLYAP06
area "Dry Receiving"
interface radio1
shutdown
channel 6
power 8
placement outdoor
antenna-gain 0.0
interface radio2
placement outdoor
interface vlan10
ip address 10.200.197.26/25
!
ap6562 FC-0A-81-17-97-28
use profile default-ap6562
use rf-domain default
hostname USLYAP18
area "Tote Receiving"
interface radio1
channel 1
power 30
placement outdoor
interface radio2
placement outdoor
interface vlan10
ip address 10.200.197.38/25
!
!
end

RE: Why are some WLANs not tunneling traffic on all access points?

Andrew_Webster — Thu, 15 Feb 2018 21:24:00 GMT

Hi Micah,
The first thing I would check are the network switches into which the RFS4000s are connected to be sure that the VLAN settings on the ports weren't lost because of the power outage.

After that, there are a number of potential "issues" with the config; here are some things to think about...
I noticed that the configuration in the RFS4000 profile seems to indicate that the RFS4000s are possibly connected into the network with more than one port. This type of connection should be avoided as it can cause ports to be shutdown unexpectedly because of spanning-tree.
A better configuration is to only use the UP1 port and have it configured with all the VLANs, including AP adoption VLAN. In your case it appears as if UP1 is only for AP adoption and GE1 is for everything else.

There are some unusual power settings on some of the APs, some are at minimum power, others at maximum, and one is even shut down. Check and review power settings to ensure that the RF signal is covering the space adequately. Consider having an "exit survey" done by a wireless professional.

I noticed that radio 2 (5GHz) isn't configured anywhere, meaning it will use smart-rf to auto channel/power. If you don't physically have 5GHz antennas connected to the AP6562, or use 5GHz wireless clients, consider shutting down the radio, as dual-band clients could see the 5GHz radio when very close by, but won't get any usability out of it.

Other troubleshooting tips
Is the cluster working properly? Check output of "show cluster members".
If the cluster becomes broken (or one of the RFSes is no longer in cluster), the license pool only lasts 100 days, after which they revert to their native license quantities. Check output of "show licenses".
Check output of "show adoption status" to ensure that all the expected APs are showing up.

RE: Why are some WLANs not tunneling traffic on all access points?

Micah — Thu, 15 Feb 2018 23:42:00 GMT

Thank you so much for all of the suggestions, Andrew!!

Some responses and additional questions:

1. I checked the switch port configurations, and they are as they should be. Identical for both controllers, and matching the controller configuration.

2. I will look for an opportunity to move the adoption VLAN onto the same port with everything else, as a best practice. I don't think spanning tree has been hurting me so far though, so I have not seen an indication in the logs of spanning tree adjustments or of ports going up/down while experiencing these issues on the network.

3. The unusual power settings are the result of a relatively long process of tweaking things to fix coverage issues in and around the building. There are a lot of concrete and metal walls with large doors that open and close, which leads to a constantly varying footprint for many of the access points. It took a lot of tuning to optimize the footprint of each AP, but we finally got it to a good place where client connectivity was stable, which was quite a while before this problem started. We did have a wireless professional help us with that, using real-time heat map measurements.

4. The issues we had that required extensive tuning only seemed to be impacting our 2.4GHz clients. We do have 5GHz clients, but we never seemed to have any connectivity issues with them, so I never messed with those radio settings. I think this is perhaps because the 5GHz band has more non-overlapping channels, so the smart-rf is better able to handle that automatically?

5. From what you say about cluster licensing, I think this may be the root of our problem. Right now the secondary controller is offline. I thought these units acted in a simple active/standby manner, and had no idea licensing would be affected when one of them is offline. We had a packet storm several months ago, and shutting down the switch ports to the standby controller resolved the issue. I haven't had an opportunity for a maintenance window to reboot it and bring it back online since then, so it has just been sitting offline for several months, possibly more than 100 days. I have a maintenance window this Sunday, so I can bring it back online then. Do you know if the cluster will automatically sort out its licensing status when the second unit comes back online, or is there something I need to do to get them synced up again on licensing?

Here's the output regarding cluster and licensing:

USLYWLAN1#show cluster configuration

Cluster Configuration Information
Name : USLYWLAN
Configured Mode : Active
Master Priority : 250
Force configured state : Disabled
Force configured state delay : 5 minutes
Handle STP : Enabled
USLYWLAN1#show cluster status

Cluster Runtime Information
Protocol version : 1
Cluster operational state : active
AP license : 12
AAP license : 12
AP count : 0
AAP count : 19
Max AP adoption capacity : 36
Number of connected member(s): 0
USLYWLAN1#show cluster members
------------------------------------------------------------------------------------------
HOSTNAME MEMBER-ID MAC MASTER OPERATIONAL-STATE LAST-SEEN
------------------------------------------------------------------------------------------
USLYWLAN1 19.DD.49.EC B4-C7-99-DD-49-EC True active self
USLYWLAN2 B4-C7-99-DD-4F-46 False down
------------------------------------------------------------------------------------------
USLYWLAN1#show licenses
Serial Number : 13158522400016

Device Licenses:
AP-LICENSE
String : DEFAULT-6AP-LICENSE
Value : 6
AAP-LICENSE
String : ***************************
Value : 12
ADVANCED-SECURITY
String : DEFAULT-ADV-SEC-LICENSE

Cluster Licenses:
AP-LICENSE
Value : 12
Used : 7
AAP-LICENSE
Value : 12
Used : 12

Active Members:
--------------------------------------------------------------------------------
MEMBER SERIAL AP LIC AAP LIC NO.APS NO.AAPS
--------------------------------------------------------------------------------
B4-C7-99-DD-49-EC 13158522400016 6 12 0 19
--------------------------------------------------------------------------------

Non-Active Members:
--------------------------------------------------------------------------------
MEMBER SERIAL AP LIC AAP LIC VALIDITY(HRS)
--------------------------------------------------------------------------------
B4-C7-99-DD-4F-46 13179522400028 6 0 1

--------------------------------------------------------------------------------
USLYWLAN1#

Thank you!!
Micah

RE: Why are some WLANs not tunneling traffic on all access points?

Micah — Thu, 15 Feb 2018 23:43:00 GMT

Also, here is the current adoption status.

USLYWLAN1#show adoption status
----------------------------------------------------------------------------------------------------------
AP-NAME VERSION CFG-STAT ADOPTED-BY LAST-ADOPTION UPTIME
----------------------------------------------------------------------------------------------------------
USLYAP21 5.4.4.0-007R configured USLYWLAN1 2017-11-09 17:43:17 97 days 20:48:42
USLYAP01 5.4.4.0-007R configured USLYWLAN1 2017-11-02 07:05:26 105 days 08:35:51
USLYAP20 5.4.4.0-007R configured USLYWLAN1 2017-11-02 07:05:30 113 days 19:54:07
USLYAP02 5.4.4.0-007R configured USLYWLAN1 2017-11-02 07:05:29 105 days 08:35:49
USLYAP04 5.4.4.0-007R configured USLYWLAN1 2017-11-06 22:08:12 100 days 16:23:11
USLYAP09 5.4.4.0-007R configured USLYWLAN1 2017-11-02 07:05:25 105 days 08:35:53
USLYAP14 5.4.4.0-007R configured USLYWLAN1 2017-11-02 07:05:31 910 days 01:23:09
USLYAP05 5.4.4.0-007R configured USLYWLAN1 2017-11-02 07:05:30 105 days 08:35:52
USLYAP16 5.4.4.0-007R configured USLYWLAN1 2017-11-02 07:05:32 910 days 01:35:15
USLYAP13 5.4.4.0-007R configured USLYWLAN1 2017-11-02 07:05:26 910 days 01:26:17
USLYAP12 5.4.4.0-007R configured USLYWLAN1 2017-11-02 07:05:36 910 days 01:28:08
USLYAP11 5.4.4.0-007R configured USLYWLAN1 2017-11-02 07:05:36 910 days 01:24:08
USLYAP10 5.4.4.0-007R configured USLYWLAN1 2017-11-02 07:05:25 105 days 08:35:53
USLYAP07 5.4.4.0-007R configured USLYWLAN1 2017-11-02 07:05:22 105 days 08:35:42
USLYAP15 5.4.4.0-007R configured USLYWLAN1 2017-11-02 07:05:26 910 days 01:29:38
USLYAP08 5.4.4.0-007R configured USLYWLAN1 2017-11-02 07:05:30 910 days 01:31:17
USLYAP17 5.4.4.0-007R configured USLYWLAN1 2017-11-02 07:05:22 113 days 20:00:59
USLYAP06 5.4.4.0-007R configured USLYWLAN1 2017-11-02 07:05:21 105 days 08:35:53
USLYAP18 5.4.4.0-007R configured USLYWLAN1 2017-11-02 07:05:21 113 days 19:39:13
------------------------------------------------------------------------------------------------------------
Total number of APs displayed: 19
USLYWLAN1#

RE: Why are some WLANs not tunneling traffic on all access points?

Andrew_Webster — Fri, 16 Feb 2018 01:50:00 GMT

Hi Micah,

It looks as if you have a total of 21 APs in your config, but only 19 APs are adopted.
Use: "show wireless ap configured" and "show adoption offline" to see which ones are missing.
On the primary RFS, You have a license for 18 APs (default 6 + 12 additional), and the default 6 APs on the secondary RFS, making a total of 24 when everything is working properly.
However, the license pool (validity period) appears to be expiring in 1 hour (so it has been 100 days), in which case you will not be able to continue to adopt all 19 APs, and one AP will be dropped, so you are about to experience more problems.

The fact that you mention that you had disabled switch ports because of a broadcast storm problem points back to the fact that the RFSes are connected in duplicate into the network...not ideal, but you'll have to live with it until you can change the topology. I'd also be very careful of the 'cluster handle-stp' that is present in your configuration.
One suggestion would be to restart the second RFS, enable the network ports, if only for a minute or two, just to re-sync the license pool until you can deal with the issue in a more permanent fashion.
You could also look at your switches' spanning tree status on the ports facing the RFSes to ensure none are in blocking or alternate.

I have seen issues with older versions of RFS4000 code (5.4.x specifically) where two cluster members would get into a shouting match with each other and send >10,000 packets/sec to the broadcast address, thus creating what appears to be a broadcast storm.
I would suggest you upgrade to a more recent firmware version, in part to stabilise the cluster and additionally to address the WPA2 KRACK vulnerability (your auditors will be happy... see: https://extremeportal.force.com/ExtrArticleDetail?n=000018005)
The upgrade should be seamless, but use the RFS4000 LEAN image and load AP firmware for AP6532 and AP6562 into the RFSes once they have been upgraded so that the APs can also be upgraded.

RE: Why are some WLANs not tunneling traffic on all access points?

Micah — Fri, 16 Feb 2018 08:06:00 GMT

Andrew,

Thank you so much for your observations!

The two access points that were already offline were known (one is a spare). You pointed out the 18/19 issue just in time though, and I disabled one additional access point that is not needed right now, so I should be safe at 18 for the moment.

During my maintenance window this weekend, I should be able to bring the secondary online to re-sync the license status, and to adjust the ports on both controllers to use just one trunk interface. I will also schedule a time to do a code upgrade, but that will probably have to wait a while longer. The shouting match scenario you mentioned seems plausible. I believe we have seen that issue twice over 4 years. The first time a reboot resolved it.

I see the 'cluster handle-stp' line in the config, but it's not familiar to me. I am seeing some references to it online, but nothing explaining what it does. Would it be better if I disabled that, and just let the switches handle stp, especially after the controllers are only on a single interface each?

Many thanks,
Micah

RE: Why are some WLANs not tunneling traffic on all access points?

Doug — Tue, 20 Feb 2018 20:00:00 GMT

Hello Micah,

Were you able to resolve this issue?

RE: Why are some WLANs not tunneling traffic on all access points?

Micah — Wed, 21 Feb 2018 01:42:00 GMT

Hi Doug, thank you for following up. I was able to repair the cluster over the weekend, but it doesn't seem to have helped with the main problem. My next step will be to upgrade the code to the latest version. It will probably take me several weeks to get another downtime window scheduled though.

RE: Why are some WLANs not tunneling traffic on all access points?

Andrew_Webster — Thu, 22 Feb 2018 23:13:00 GMT

Extreme has just written an article specifically dealing with licenses across clusters.
https://extremeportal.force.com/ExtrArticleDetail?n=000021938