topic Re: WiNG vx9000 not clustering since being relocated to new HyperV host in ExtremeWireless (WiNG)

WiNG vx9000 not clustering since being relocated to new HyperV host

Eric_Burke — Tue, 10 Mar 2020 00:42:35 GMT

We recently deployed a pair of vx9000’s in HyperV. The controllers are each on a different host. When we started building, we only had one host ready so we built the pair on that box. Now that both hosts are up, we exported and imported one of the units, Same IP address, just a new host. We found that mistakenly we didn’t clone the MAC and lost the licensing. We then set the MAC back, and confirmed that the vx9000’s coudl still ping eachother. They’re sending hello’s via MINT. A support engineer helped us to realize that the license got dropped, presumably when getting the wrong MAC, so we re-licensed it. Still, we could not get the cluster back up. The engineer dropped the node we’d moved and manually rejoined the cluster. This seemed to work for a brief couple of minutes, but then we kept seeing the active role shifting back and forth between controllers. On the relocated unit, we’re seeing the following message:

CMASTER_CFG_UPDATE_FAIL: Cluster Master Rejecting Config from unknown host. Not sure what else we need to do to resolve the issue? Any guidance?

Re: WiNG vx9000 not clustering since being relocated to new HyperV host

Christoph_S — Tue, 10 Mar 2020 00:52:53 GMT

Hello Eric,

Try to disable and recreate the cluster using these instructions:

Reestablish broken cluster

Thank you,

Chris

Re: WiNG vx9000 not clustering since being relocated to new HyperV host

ckelly — Tue, 10 Mar 2020 01:01:44 GMT

Eric,

Can you give the output for the following:

(On BOTH cluster member, run these)

show cluster status

show cluster configuration

show cluster members

show cluster history

Then, on the Primary cluster member, let’s try some debugging: (let run long enough to see cluster related messages) The output should hopefully reveal a little more of what’s happening here.

# debug cfgd cluster

# logging monitor debugging

Re: WiNG vx9000 not clustering since being relocated to new HyperV host

Eric_Burke — Tue, 10 Mar 2020 02:02:42 GMT

Here is the WEST controller (the one that stayed up, is primary, never moved):

VX-WEST*#show cluster status Cluster Runtime Information Protocol version : 1 Cluster operational state : active AP license : 0 AAP license : 128 AP count : 0 AAP count : 0 Max AP adoption capacity : 25600 Number of connected member(s): 0 VX-WEST*# VX-WEST*#show cluster configuration Cluster Configuration Information Name : HeathVillage Configured Mode : Active Master Priority : 255 Force configured state : Disabled Force configured state delay : 5 minutes Handle STP : Disabled Radius Counter DB Sync Time : 5 minutes VX-WEST*# VX-WEST*#show cluster members ------------------------------------------------------------------------------------------ HOSTNAME MEMBER-ID MAC MASTER OPERATIONAL-STATE LAST-SEEN ------------------------------------------------------------------------------------------ VX-WEST* 12.40.AD.08 00-15-5D-40-AD-08 True active self VX-EAST 00-15-5D-40-AD-09 False down ------------------------------------------------------------------------------------------ VX-WEST*# VX-WEST*#show cluster history ----------------------------------------------------------------------------------------------------------------------------- HOSTNAME MEMBER-ID EVENT TIME-STAMP REASON ----------------------------------------------------------------------------------------------------------------------------- VX-WEST 12.40.AD.08 start-election 2020-03-09 19:31:34 Member 12.40.AD.09 joined VX-WEST 12.40.AD.08 is-active 2020-03-09 18:20:32 Primary down, currently active, stay active VX-WEST 12.40.AD.08 start-election 2020-03-09 18:20:02 Member 12.40.AD.09 down VX-WEST 12.40.AD.08 start-config-sync 2020-03-09 18:19:11 Member 12.40.AD.08 won election and is the cluster master VX-WEST 12.40.AD.08 start-election 2020-03-09 18:18:36 Member 12.40.AD.09 joined VX-WEST 12.40.AD.08 is-active 2020-03-09 18:18:14 Primary down, currently active, stay active VX-WEST 12.40.AD.08 start-election 2020-03-09 18:17:44 Member 12.40.AD.09 down VX-WEST 12.40.AD.08 is-active 2020-03-09 18:16:51 No configured standby members, stay active VX-WEST 12.40.AD.08 handle-member-update 2020-03-09 18:16:36 Member 12.40.AD.09 state change to active VX-WEST 12.40.AD.08 config-update-success 2020-03-09 18:14:43 Successfully applied config VX-WEST 12.40.AD.08 is-active 2020-03-09 18:14:36 Primary down, currently active, stay active VX-WEST 12.40.AD.08 config-update-rejected 2020-03-09 18:14:36 ['rejecting config from unknown host'] VX-WEST 12.40.AD.08 start-config-sync 2020-03-09 18:14:36 Member 12.40.AD.08 won election and is the cluster master VX-WEST 12.40.AD.08 start-election 2020-03-09 18:14:06 Member 12.40.AD.09 joined VX-WEST 12.40.AD.08 is-active 2020-03-09 18:10:49 Primary down, currently active, stay active VX-WEST 12.40.AD.08 start-election 2020-03-09 18:10:19 Member 12.40.AD.09 down VX-WEST 12.40.AD.08 is-active 2020-03-09 18:07:21 No configured standby members, stay active VX-WEST 12.40.AD.08 config-update-success 2020-03-09 18:07:06 Successfully applied config VX-WEST 12.40.AD.08 handle-member-update 2020-03-09 18:07:06 Member 12.40.AD.09 state change to active VX-WEST 12.40.AD.08 is-active 2020-03-09 18:07:06 Primary down, currently active, stay active VX-WEST 12.40.AD.08 start-config-sync 2020-03-09 18:07:06 Member 12.40.AD.08 won election and is the cluster master VX-WEST 12.40.AD.08 start-election 2020-03-09 18:06:35 Member 12.40.AD.09 joined VX-WEST 12.40.AD.08 is-active 2020-03-09 18:05:11 Primary down, currently active, stay active VX-WEST 12.40.AD.08 start-election 2020-03-09 18:03:59 Member 12.40.AD.09 joined VX-WEST 12.40.AD.08 is-active 2020-03-09 18:01:52 Primary down, currently active, stay active VX-WEST 12.40.AD.08 start-election 2020-03-09 18:01:22 Member 12.40.AD.09 down VX-WEST 12.40.AD.08 is-active 2020-03-09 18:00:21 No configured standby members, stay active VX-WEST 12.40.AD.08 config-update-success 2020-03-09 18:00:13 Successfully applied config VX-WEST 12.40.AD.08 handle-member-update 2020-03-09 18:00:06 Member 12.40.AD.09 state change to active VX-WEST 12.40.AD.08 config-update-rejected 2020-03-09 18:00:05 ['rejecting config from unknown host'] VX-WEST 12.40.AD.08 is-active 2020-03-09 18:00:05 Primary down, currently active, stay active VX-WEST 12.40.AD.08 start-config-sync 2020-03-09 18:00:05 Member 12.40.AD.08 won election and is the cluster master VX-WEST 12.40.AD.08 start-election 2020-03-09 17:59:35 Member 12.40.AD.09 joined VX-WEST 12.40.AD.08 is-active 2020-03-09 17:59:19 Primary down, currently active, stay active VX-WEST 12.40.AD.08 start-election 2020-03-09 17:58:49 Member 12.40.AD.09 down VX-WEST 12.40.AD.08 is-active 2020-03-09 17:58:21 No configured standby members, stay active VX-WEST 12.40.AD.08 config-update-success 2020-03-09 17:58:13 Successfully applied config VX-WEST 12.40.AD.08 handle-member-update 2020-03-09 17:58:06 Member 12.40.AD.09 state change to active VX-WEST 12.40.AD.08 is-active 2020-03-09 17:58:06 Primary down, currently active, stay active VX-WEST 12.40.AD.08 config-update-rejected 2020-03-09 17:58:05 ['rejecting config from unknown host'] VX-WEST 12.40.AD.08 start-config-sync 2020-03-09 17:58:05 Member 12.40.AD.08 won election and is the cluster master VX-WEST 12.40.AD.08 start-election 2020-03-09 17:57:35 Member 12.40.AD.09 joined VX-WEST 12.40.AD.08 is-active 2020-03-09 17:56:58 Primary down, currently active, stay active VX-WEST 12.40.AD.08 start-election 2020-03-09 17:55:44 Member 12.40.AD.09 down VX-WEST 12.40.AD.08 config-update-success 2020-03-09 17:54:36 Successfully applied config VX-WEST 12.40.AD.08 is-active 2020-03-09 17:54:35 Primary down, currently active, stay active -----------------------------------------------------------------------------------------------------------------------------

Here is the one we moved, rejoined to the cluster after fixing the license issue and setting the MAC back to the original address:

VX-EAST>show cluster status Cluster Runtime Information Protocol version : 1 Cluster operational state : standby AP license : 0 AAP license : 128 AP count : 0 AAP count : 0 Max AP adoption capacity : 25600 Number of connected member(s): 0 VX-EAST> VX-EAST>show cluster configuration Cluster Configuration Information Name : HeathVillage Configured Mode : Active Master Priority : 128 Force configured state : Disabled Force configured state delay : 5 minutes Handle STP : Disabled Radius Counter DB Sync Time : 5 minutes VX-EAST> VX-EAST>show cluster members Cluster master election in progress Configured cluster members 00-15-5D-40-AD-08 00-15-5D-40-AD-09 VX-EAST> VX-EAST>show cluster history ----------------------------------------------------------------------------------------------------------------------------- HOSTNAME MEMBER-ID EVENT TIME-STAMP REASON ----------------------------------------------------------------------------------------------------------------------------- VX-EAST 12.40.AD.09 start-election 2020-03-09 19:02:09 Member 12.40.AD.08 joined VX-EAST 12.40.AD.09 is-active 2020-03-09 19:01:41 Primary down, currently active, stay active VX-EAST 12.40.AD.09 start-election 2020-03-09 19:01:11 Member 12.40.AD.08 down VX-EAST 12.40.AD.09 config-update-success 2020-03-09 18:59:12 Successfully applied config VX-EAST 12.40.AD.09 is-active 2020-03-09 18:57:39 Primary down, currently standby, become active VX-EAST 12.40.AD.09 start-election 2020-03-09 18:56:11 Member 12.40.AD.09 joined VX-EAST 12.40.AD.09 start-election 2020-03-09 17:28:23 Member 12.40.AD.08 down VX-EAST 12.40.AD.09 config-update-success 2020-03-09 17:25:24 Successfully applied config VX-EAST 12.40.AD.09 start-election 2020-03-09 17:23:50 Member 12.40.AD.08 joined VX-EAST 12.40.AD.09 is-active 2020-03-09 17:23:04 Primary down, currently active, stay active VX-EAST 12.40.AD.09 start-election 2020-03-09 17:21:50 Member 12.40.AD.08 joined VX-EAST 12.40.AD.09 is-active 2020-03-09 17:21:33 Primary down, currently active, stay active VX-EAST 12.40.AD.09 start-election 2020-03-09 17:21:03 Member 12.40.AD.08 down VX-EAST 12.40.AD.09 config-update-success 2020-03-09 17:17:53 Successfully applied config VX-EAST 12.40.AD.09 start-election 2020-03-09 17:16:35 Member 12.40.AD.08 joined VX-EAST 12.40.AD.09 is-active 2020-03-09 17:15:59 Primary down, currently active, stay active VX-EAST 12.40.AD.09 start-election 2020-03-09 17:14:20 Member 12.40.AD.08 joined VX-EAST 12.40.AD.09 is-active 2020-03-09 17:12:43 Primary down, currently active, stay active VX-EAST 12.40.AD.09 start-election 2020-03-09 17:12:13 Member 12.40.AD.08 down VX-EAST 12.40.AD.09 config-update-success 2020-03-09 17:11:05 Successfully applied config VX-EAST 12.40.AD.09 start-election 2020-03-09 17:09:41 Member 12.40.AD.08 down VX-EAST 12.40.AD.09 config-update-success 2020-03-09 17:09:06 Successfully applied config VX-EAST 12.40.AD.09 start-election 2020-03-09 17:06:39 Member 12.40.AD.08 down VX-EAST 12.40.AD.09 config-update-success 2020-03-09 17:05:32 Successfully applied config VX-EAST 12.40.AD.09 start-election 2020-03-09 17:03:36 Member 12.40.AD.08 joined VX-EAST 12.40.AD.09 is-active 2020-03-09 17:03:06 Primary down, currently active, stay active VX-EAST 12.40.AD.09 start-election 2020-03-09 17:02:36 Member 12.40.AD.08 down VX-EAST 12.40.AD.09 config-update-success 2020-03-09 17:00:18 Successfully applied config VX-EAST 12.40.AD.09 start-election 2020-03-09 16:59:06 Member 12.40.AD.08 joined VX-EAST 12.40.AD.09 is-active 2020-03-09 16:59:04 Primary down, currently active, stay active VX-EAST 12.40.AD.09 start-election 2020-03-09 16:57:51 Member 12.40.AD.08 joined VX-EAST 12.40.AD.09 is-active 2020-03-09 16:57:48 Primary down, currently active, stay active VX-EAST 12.40.AD.09 start-election 2020-03-09 16:54:31 Member 12.40.AD.08 down VX-EAST 12.40.AD.09 config-update-success 2020-03-09 16:52:48 Successfully applied config VX-EAST 12.40.AD.09 start-election 2020-03-09 16:50:43 Member 12.40.AD.08 down VX-EAST 12.40.AD.09 start-election 2020-03-09 16:47:37 Member 12.40.AD.08 joined VX-EAST 12.40.AD.09 is-active 2020-03-09 16:47:26 Primary down, currently active, stay active VX-EAST 12.40.AD.09 start-election 2020-03-09 16:45:25 Member 12.40.AD.08 down VX-EAST 12.40.AD.09 config-update-success 2020-03-09 16:42:33 Successfully applied config VX-EAST 12.40.AD.09 start-election 2020-03-09 16:41:53 Member 12.40.AD.08 joined VX-EAST 12.40.AD.09 is-active 2020-03-09 16:37:49 Primary down, currently active, stay active VX-EAST 12.40.AD.09 start-election 2020-03-09 16:37:19 Member 12.40.AD.08 down VX-EAST 12.40.AD.09 config-update-success 2020-03-09 16:36:29 Successfully applied config VX-EAST 12.40.AD.09 start-election 2020-03-09 16:34:23 Member 12.40.AD.08 joined VX-EAST 12.40.AD.09 is-active 2020-03-09 16:34:17 Primary down, currently active, stay active VX-EAST 12.40.AD.09 start-election 2020-03-09 16:32:24 Member 12.40.AD.08 joined VX-EAST 12.40.AD.09 is-active 2020-03-09 16:32:01 Primary down, currently active, stay active VX-EAST 12.40.AD.09 start-election 2020-03-09 16:31:31 Member 12.40.AD.08 down VX-EAST 12.40.AD.09 config-update-success 2020-03-09 16:31:25 Successfully applied config VX-EAST 12.40.AD.09 start-election 2020-03-09 16:30:23 Member 12.40.AD.08 joined -----------------------------------------------------------------------------------------------------------------------------

Debug coming in next...

Re: WiNG vx9000 not clustering since being relocated to new HyperV host

Eric_Burke — Tue, 10 Mar 2020 02:07:40 GMT

Here’s the debug from the “primary” controller:

VX-WEST*>en VX-WEST*#debug cfgd cluster VX-WEST*#logging monitor debugging VX-WEST*#Mar 09 19:36:48 2020: USER: main.pyo: received packet START_ELECTION ver 1 prio 128 master period 0 rank 220 dc 1 id 12.40.AD.09 Mar 09 19:36:48 2020: USER: main.pyo: received packet from unknown member START_ELECTION ver 1 prio 128 master period 0 rank 220 dc 1 id 12.40.AD.09 Mar 09 19:36:48 2020: USER: main.pyo: received packet CANDIDATE ver 1 prio 128 master period 0 rank 220 dc 1 id 12.40.AD.09 Mar 09 19:36:48 2020: USER: main.pyo: received packet from unknown member CANDIDATE ver 1 prio 128 master period 0 rank 220 dc 1 id 12.40.AD.09 Mar 09 19:36:49 2020: USER: main.pyo: election recv timeout Mar 09 19:36:49 2020: USER: main.pyo: completed election master is 12.40.AD.08 Mar 09 19:36:49 2020: USER: main.pyo: new member 12.40.AD.08 Mar 09 19:36:49 2020: USER: main.pyo: adding device 12.40.AD.08 [00-15-5D-40-AD-08] as existing cluster member Mar 09 19:36:49 2020: USER: main.pyo: check if cluster state transition is required, mcount 1 member_id 00.00.00.00 member_changed_state 0 Mar 09 19:36:49 2020: USER: main.pyo: active cluster member is down; standby need to become active Mar 09 19:36:49 2020: USER: main.pyo: publish own cluster status=[active] internally Mar 09 19:36:49 2020: VX-WEST : %DIAG-6-NEW_LED_STATE: LED state message LED_ACTIVE_ADOPTING from module CFGD Mar 09 19:36:52 2020: USER: main.pyo: publish own cluster status=[active] internally Mar 09 19:36:52 2020: USER: main.pyo: handle_cluster_member_update members: ['12.40.AD.08', '12.40.AD.09'] Mar 09 19:36:52 2020: USER: main.pyo: handle_cluster_member_update new members, start pending election tasklet cluster_member_update_count=796 Mar 09 19:36:52 2020: USER: main.pyo: pending_election_tasklet(): cluster_member_update_count=796 member_update_count=796 members=['12.40.AD.08', '12.40.AD.09'] Mar 09 19:37:02 2020: USER: main.pyo: pending_election_tasklet(): after hysteresis delay cluster_member_update_count=796 member_update_count=796 Mar 09 19:37:02 2020: USER: main.pyo: start_election(): the_election.candidate_members: ['12.40.AD.08'] members: ['12.40.AD.08', '12.40.AD.09'] Mar 09 19:37:02 2020: USER: main.pyo: updating new members Mar 09 19:37:02 2020: USER: main.pyo: Updating new members ['12.40.AD.09'] Mar 09 19:37:02 2020: USER: main.pyo: created new local pkt: MASTER_UPDATE ver 1 prio 255 master period 57551 rank 220 dc 0 id 12.40.AD.08 Mar 09 19:37:02 2020: USER: main.pyo: dequeue - need to run election Mar 09 19:37:02 2020: USER: main.pyo: starting election forced=False immediate_master_election=False Mar 09 19:37:02 2020: USER: main.pyo: created new local pkt: CANDIDATE ver 1 prio 255 master period 57551 rank 220 dc 1 id 12.40.AD.08 Mar 09 19:37:02 2020: USER: main.pyo: created new local pkt: START_ELECTION ver 1 prio 255 master period 57551 rank 220 dc 1 id 12.40.AD.08 Mar 09 19:37:02 2020: USER: main.pyo: broadcasting election start Mar 09 19:37:02 2020: USER: main.pyo: announcing self as candidate Mar 09 19:37:02 2020: USER: main.pyo: cluster reset() member count 1 connected member count 1 Mar 09 19:37:02 2020: USER: main.pyo: adding device 12.40.AD.08 [00-15-5D-40-AD-08] as new cluster member Mar 09 19:37:02 2020: USER: main.pyo: canceling ask_msg_update_timer Mar 09 19:37:02 2020: VX-WEST : %DIAG-6-NEW_LED_STATE: LED state message LED_ACTIVE_ADOPTING from module CFGD Mar 09 19:37:02 2020: USER: main.pyo: received packet CANDIDATE ver 1 prio 128 master period 0 rank 220 dc 1 id 12.40.AD.09 Mar 09 19:37:02 2020: USER: main.pyo: Local election score improved, 2 known configs Mar 09 19:37:02 2020: USER: main.pyo: created new local pkt: CANDIDATE ver 1 prio 255 master period 57551 rank 220 dc 2 id 12.40.AD.08 Mar 09 19:37:02 2020: USER: main.pyo: best set to 12.40.AD.08 (CANDIDATE ver 1 prio 255 master period 57551 rank 220 dc 2 id 12.40.AD.08) Mar 09 19:37:02 2020: USER: main.pyo: retaining old candidate 12.40.AD.08 as best, not 12.40.AD.09 Mar 09 19:37:02 2020: USER: main.pyo: received packet CANDIDATE ver 1 prio 128 master period 0 rank 220 dc 2 id 12.40.AD.09 Mar 09 19:37:02 2020: USER: main.pyo: retaining old candidate 12.40.AD.08 as best, not 12.40.AD.09 Mar 09 19:37:17 2020: USER: main.pyo: election recv timeout Mar 09 19:37:17 2020: USER: main.pyo: Announcing self as master Mar 09 19:37:17 2020: USER: main.pyo: created new local pkt: ELECTION_WINNER ver 1 prio 255 master period 57566 rank 220 dc 2 id 12.40.AD.08 Mar 09 19:37:18 2020: VX-WEST : %SYSTEM-5-UI_USER_AUTH_SUCCESS: UI user 'admin' from: '10.66.64.225' authentication successful Mar 09 19:37:20 2020: USER: main.pyo: ClusterGenerator: klist=['00-15-5D-40-AD-08', '00-15-5D-40-AD-09'] Mar 09 19:37:31 2020: USER: main.pyo: got cluster msg 3 from 12.40.AD.09 (306228489, 0, ('i', 24576, '10.200.247.192')) Mar 09 19:37:31 2020: USER: main.pyo: Received member update msg 3 from unknown member 12.40.AD.09 (306228489, 0, ('i', 24576, '10.200.247.192')) Mar 09 19:37:31 2020: USER: main.pyo: got cluster msg 3 from 12.40.AD.09 (306228489, 0, ('i', 24576, '10.200.247.192')) Mar 09 19:37:31 2020: USER: main.pyo: Received member update msg 3 from unknown member 12.40.AD.09 (306228489, 0, ('i', 24576, '10.200.247.192')) Mar 09 19:37:31 2020: VX-EAST : %DIAG-6-NEW_LED_STATE: LED state message LED_ACTIVE_ADOPTING from module CFGD Mar 09 19:37:32 2020: USER: main.pyo: election recv timeout Mar 09 19:37:32 2020: USER: main.pyo: completed election master is 12.40.AD.08 Mar 09 19:37:32 2020: USER: main.pyo: new member 12.40.AD.08 Mar 09 19:37:32 2020: USER: main.pyo: adding device 12.40.AD.08 [00-15-5D-40-AD-08] as existing cluster member Mar 09 19:37:32 2020: USER: main.pyo: new member 12.40.AD.09 Mar 09 19:37:32 2020: USER: main.pyo: adding device 12.40.AD.09 [00-15-5D-40-AD-09] as new cluster member Mar 09 19:37:32 2020: USER: main.pyo: mint_id = 12.40.AD.09 APLIC_INSTALLED=0 APLIC_ACQUIRED=0 APLIC_GRANTED=0 APLIC_PENDING=0 APLIC_TOTAL=0 Mar 09 19:37:32 2020: USER: main.pyo: Mar 09 19:37:32 2020: USER: main.pyo: mint_id = 12.40.AD.09 AAPLIC_INSTALLED=64 AAPLIC_ACQUIRED=0 AAPLIC_GRANTED=0 AAPLIC_PENDING=0 AAPLIC_TOTAL=64 Mar 09 19:37:32 2020: USER: main.pyo: Mar 09 19:37:32 2020: USER: main.pyo: add_member() send update message to member 12.40.AD.09 Mar 09 19:37:32 2020: USER: main.pyo: send update req message to member 12.40.AD.09 Mar 09 19:37:32 2020: USER: main.pyo: master-updater woke up - 1 changes in the list Mar 09 19:37:32 2020: USER: main.pyo: tx_update CS_ACT_CFG_FULL - revision 3 Mar 09 19:37:32 2020: USER: main.pyo: tx_update member rev is 0 Mar 09 19:37:32 2020: USER: main.pyo: tx-update send update raw to 12.40.AD.09 Mar 09 19:37:32 2020: USER: main.pyo: sending UPDATE revision: 3 hdr size: 20, data size: 50539 Mar 09 19:37:32 2020: USER: main.pyo: CS_Send() connecting to 12.40.AD.09 Mar 09 19:37:32 2020: VX-WEST : %DIAG-6-NEW_LED_STATE: LED state message LED_ACTIVE_ADOPTING from module CFGD Mar 09 19:37:32 2020: USER: main.pyo: CS_Send() connected Mar 09 19:37:33 2020: VX-EAST : %CFGD-6-ACL_RULE_ALTERED: USER: cfgd session 13: ACL BROADCAST-MULTICAST-CONTROL rule is getting altered Mar 09 19:37:33 2020: VX-EAST : %CFGD-6-ACL_RULE_ALTERED: USER: cfgd session 13: ACL PERMIT-ARP-AND-IPv4 rule is getting altered Mar 09 19:37:33 2020: VX-EAST : %SYSTEM-6-CONFIG_COMMIT: Configuration commit by user 'cfgd' (update own config) from '127.0.0.1' Mar 09 19:37:33 2020: USER: main.pyo: cluster config update OK, current revision: 3, expected revision: 3 Mar 09 19:37:33 2020: USER: main.pyo: tx_update CS_ACT_CFG_FULL - revision 3 host 12.40.AD.09 is OK Mar 09 19:37:33 2020: VX-EAST : %SYSTEM-6-CONFIG_REVISION: Configuration revision updated to 3 from 3 Mar 09 19:37:58 2020: VX-WEST : %SYSTEM-5-UI_USER_AUTH_SUCCESS: UI user 'admin' from: '10.66.64.225' authentication successful Mar 09 19:38:01 2020: USER: main.pyo: ClusterGenerator: klist=['00-15-5D-40-AD-08', '00-15-5D-40-AD-09'] Mar 09 19:38:04 2020: USER: main.pyo: publish own cluster status=[active] internally Mar 09 19:38:04 2020: USER: main.pyo: handle_cluster_member_update members: ['12.40.AD.08'] Mar 09 19:38:04 2020: USER: main.pyo: handle_cluster_member_update fewer members cluster_member_update_count=797 Mar 09 19:38:04 2020: USER: main.pyo: start_election(): the_election.candidate_members: ['12.40.AD.08', '12.40.AD.09'] members: ['12.40.AD.08'] Mar 09 19:38:04 2020: USER: main.pyo: no change in master, member list changed Mar 09 19:38:04 2020: USER: main.pyo: dequeue - need to run election Mar 09 19:38:04 2020: USER: main.pyo: starting election forced=False immediate_master_election=False Mar 09 19:38:04 2020: USER: main.pyo: created new local pkt: CANDIDATE ver 1 prio 255 master period 57614 rank 220 dc 1 id 12.40.AD.08 Mar 09 19:38:04 2020: USER: main.pyo: created new local pkt: START_ELECTION ver 1 prio 255 master period 57614 rank 220 dc 1 id 12.40.AD.08 Mar 09 19:38:04 2020: USER: main.pyo: broadcasting election start Mar 09 19:38:04 2020: USER: main.pyo: announcing self as candidate Mar 09 19:38:04 2020: USER: main.pyo: cluster reset() member count 2 connected member count 2 Mar 09 19:38:04 2020: USER: main.pyo: adding device 12.40.AD.08 [00-15-5D-40-AD-08] as new cluster member Mar 09 19:38:04 2020: USER: main.pyo: canceling ask_msg_update_timer Mar 09 19:38:04 2020: VX-WEST : %DIAG-6-NEW_LED_STATE: LED state message LED_ACTIVE_ADOPTING from module CFGD Mar 09 19:38:19 2020: USER: main.pyo: election recv timeout Mar 09 19:38:19 2020: USER: main.pyo: Announcing self as master Mar 09 19:38:19 2020: USER: main.pyo: created new local pkt: ELECTION_WINNER ver 1 prio 255 master period 57629 rank 220 dc 1 id 12.40.AD.08 Mar 09 19:38:34 2020: VX-WEST : %DIAG-6-NEW_LED_STATE: LED state message LED_ACTIVE_ADOPTING from module CFGD Mar 09 19:38:34 2020: USER: main.pyo: election recv timeout Mar 09 19:38:34 2020: USER: main.pyo: completed election master is 12.40.AD.08 Mar 09 19:38:34 2020: USER: main.pyo: new member 12.40.AD.08 Mar 09 19:38:34 2020: USER: main.pyo: adding device 12.40.AD.08 [00-15-5D-40-AD-08] as existing cluster member Mar 09 19:38:34 2020: USER: main.pyo: check if cluster state transition is required, mcount 1 member_id 00.00.00.00 member_changed_state 0 Mar 09 19:38:34 2020: USER: main.pyo: active cluster member is down; standby need to become active Mar 09 19:38:34 2020: USER: main.pyo: publish own cluster status=[active] internally Mar 09 19:38:54 2020: USER: main.pyo: publish own cluster status=[active] internally Mar 09 19:38:54 2020: USER: main.pyo: handle_cluster_member_update members: ['12.40.AD.08', '12.40.AD.09'] Mar 09 19:38:54 2020: USER: main.pyo: handle_cluster_member_update new members, start pending election tasklet cluster_member_update_count=798 Mar 09 19:38:54 2020: USER: main.pyo: pending_election_tasklet(): cluster_member_update_count=798 member_update_count=798 members=['12.40.AD.08', '12.40.AD.09'] Mar 09 19:38:56 2020: USER: main.pyo: publish own cluster status=[active] internally Mar 09 19:38:56 2020: USER: main.pyo: handle_cluster_member_update members: ['12.40.AD.08'] Mar 09 19:38:56 2020: USER: main.pyo: handle_cluster_member_update fewer members cluster_member_update_count=799 Mar 09 19:39:04 2020: USER: main.pyo: Peer 12.40.AD.09 connected Mar 09 19:39:04 2020: USER: main.pyo: received pkt MASTER_UPDATE ver 1 prio 128 master period 30 rank 220 dc 0 id 12.40.AD.09 Mar 09 19:39:04 2020: USER: main.pyo: Received packet from unknown member MASTER_UPDATE ver 1 prio 128 master period 30 rank 220 dc 0 id 12.40.AD.09 Mar 09 19:39:04 2020: USER: main.pyo: received pkt START_ELECTION ver 1 prio 128 master period 30 rank 220 dc 1 id 12.40.AD.09 Mar 09 19:39:04 2020: USER: main.pyo: Received packet from unknown member START_ELECTION ver 1 prio 128 master period 30 rank 220 dc 1 id 12.40.AD.09 Mar 09 19:39:04 2020: USER: main.pyo: received pkt CANDIDATE ver 1 prio 128 master period 30 rank 220 dc 1 id 12.40.AD.09 Mar 09 19:39:04 2020: USER: main.pyo: Received packet from unknown member CANDIDATE ver 1 prio 128 master period 30 rank 220 dc 1 id 12.40.AD.09 Mar 09 19:39:04 2020: USER: main.pyo: pending_election_tasklet(): after hysteresis delay cluster_member_update_count=799 member_update_count=798 Mar 09 19:39:04 2020: USER: main.pyo: do not start election due to member list change: cluster_member_update_count=799 member_update_count=798 Mar 09 19:39:09 2020: USER: main.pyo: Ignore Exception Mar 09 19:39:19 2020: USER: main.pyo: received pkt START_ELECTION ver 1 prio 128 master period 45 rank 220 dc 1 id 12.40.AD.09 Mar 09 19:39:19 2020: USER: main.pyo: Received packet from unknown member START_ELECTION ver 1 prio 128 master period 45 rank 220 dc 1 id 12.40.AD.09 Mar 09 19:39:19 2020: USER: main.pyo: received pkt CANDIDATE ver 1 prio 128 master period 45 rank 220 dc 1 id 12.40.AD.09 Mar 09 19:39:19 2020: USER: main.pyo: Received packet from unknown member CANDIDATE ver 1 prio 128 master period 45 rank 220 dc 1 id 12.40.AD.09 Mar 09 19:39:34 2020: USER: main.pyo: received pkt START_ELECTION ver 1 prio 128 master period 60 rank 220 dc 1 id 12.40.AD.09 Mar 09 19:39:34 2020: USER: main.pyo: Received packet from unknown member START_ELECTION ver 1 prio 128 master period 60 rank 220 dc 1 id 12.40.AD.09 Mar 09 19:39:34 2020: USER: main.pyo: received pkt CANDIDATE ver 1 prio 128 master period 60 rank 220 dc 1 id 12.40.AD.09 Mar 09 19:39:34 2020: USER: main.pyo: Received packet from unknown member CANDIDATE ver 1 prio 128 master period 60 rank 220 dc 1 id 12.40.AD.09 Mar 09 19:39:49 2020: USER: main.pyo: received pkt START_ELECTION ver 1 prio 128 master period 75 rank 220 dc 1 id 12.40.AD.09 Mar 09 19:39:49 2020: USER: main.pyo: Received packet from unknown member START_ELECTION ver 1 prio 128 master period 75 rank 220 dc 1 id 12.40.AD.09 Mar 09 19:39:49 2020: USER: main.pyo: received pkt CANDIDATE ver 1 prio 128 master period 75 rank 220 dc 1 id 12.40.AD.09 Mar 09 19:39:49 2020: USER: main.pyo: Received packet from unknown member CANDIDATE ver 1 prio 128 master period 75 rank 220 dc 1 id 12.40.AD.09 exit

Re: WiNG vx9000 not clustering since being relocated to new HyperV host

Matt_Unger — Tue, 10 Mar 2020 02:21:49 GMT

I am working with Eric on this issue and would like to let you know what I am seeing on the web interface for these controllers.

From the West:

From the East

Re: WiNG vx9000 not clustering since being relocated to new HyperV host

ckelly — Tue, 10 Mar 2020 02:40:01 GMT

Matt/Eric,

Looking at the GUI entries on the controller, you have the same IP address listed as using both MINT level-1 and level-2 (10.200.247.192). This is not correct and will certainly cause issues. You should only have a singe MiNT routing level defined here for both entries for cluster members. This MINT level should match the MINT level that your APs are using for adoption (this is important!)

Which controller is 10.200.247.192? (East or West?)

Which controller is 10.66.64.192? (East or West?)

From the CLI queries, things appear normal except that from the West controller’s perspective, the standby controller (East is down). This would normally be due to a loss of MINT connectivity.

From the East controller’s perspective, it’s undergoing an election process.

It boils down to the controller’s not properly communicating with each other and each seems to be thinking that the other controller is down...

I haven’t had time to sift through the debug output yet (formatting was lost and makes it difficult to read).

For now, you need to get the cluster member setup corrected (with correct IP addresses and MINT level)

Re: WiNG vx9000 not clustering since being relocated to new HyperV host

Matt_Unger — Tue, 10 Mar 2020 02:41:45 GMT

10.66.64.192 is West

10.200.247.192 is East

Re: WiNG vx9000 not clustering since being relocated to new HyperV host

Eric_Burke — Tue, 10 Mar 2020 02:53:41 GMT

@Chris Kelly ,

We’re just getting into our first more “complex” deployment of WiNG here, so it may be an issue related to our intended design. Here’s the plan we discussed with pro services. The customer has 2 campuses, connected via fiber. In each campus, we’re running a HyperV host with a vx9000. The AP’s in each campus are adopted at layer 2 in their own, unique provisioning vlans. The controllers themselves are in separate L3 VLAN’s (as two different sites), connected by firewalls that have both point to point (over fiber) and VPN backup connections between them. Our understanding was that we could cluster the controllers at Layer 3 and should we lose a HyperV host, the AP’s connected to that associated controller could use the remote site’s controller as a backup. Is that incorrect? I think that means we have controllers with MINT2 links, but AP’s with site-specific MINT1...

Re: WiNG vx9000 not clustering since being relocated to new HyperV host

ckelly — Tue, 10 Mar 2020 02:58:54 GMT

Matt,

Can you run this query:

# show mint stats

# show mint info

Your setup in the GUI should look like this...except I don’t know what your MINT routing level needs to be. I’m hoping to determine this from the output of the above queries.

Re: WiNG vx9000 not clustering since being relocated to new HyperV host

ckelly — Tue, 10 Mar 2020 03:07:39 GMT

Eric,

To confirm, the APs are adopting to the controllers at layer-2? The APs don’t have IP addresses on their management interfaces?

Controllers in a cluster being co-located is perfectly normal. How you described the intended fail-over is a common setup and is normal operation.

As for MINT though, you need to ensure that however your APs are adopting (the MINT level they’re using) this also needs to be the MINT level that the controllers are setup to use for their clustering.

E.g., you don’t/can’t have APs adopting using level-1 MINT and then have your controllers clustered using MINT level-2. They need to match.

Re: WiNG vx9000 not clustering since being relocated to new HyperV host

Eric_Burke — Tue, 10 Mar 2020 03:08:10 GMT

West:

VX-WEST*#sho mint stats
0 Level-1 neighbors
Level-1 LSP DB size 1 LSPs (1 KB)
Last Level-1 SPFs took 0.000s
Level-1 SPF (re)calculated 1638 times.
1 Level-1 paths.
1 Level-2 neighbors
Level-2 LSP DB size 1 LSPs (1 KB)
Last Level-2 SPFs took 0.000s
Level-2 SPF (re)calculated 9136 times.
1 Level-2 paths.
VX-WEST*#sho mint info
Mint ID: 12.40.AD.08
0 Level-1 neighbors
Level-1 LSP DB size 1 LSPs (1 KB)
1 Level-2 neighbors
Level-2 LSP DB size 1 LSPs (1 KB)
Level-2 gateway is unreachable
Reachable adopters: 12.40.AD.08
Max level-1 path cost: 0 (to 12.40.AD.08)

EAST:

VX-EAST#sho mint stats
0 Level-1 neighbors
Level-1 LSP DB size 1 LSPs (1 KB)
Last Level-1 SPFs took 0.000s
Level-1 SPF (re)calculated 31 times.
1 Level-1 paths.
1 Level-2 neighbors
Level-2 LSP DB size 2 LSPs (1 KB)
Last Level-2 SPFs took 0.000s
Level-2 SPF (re)calculated 61 times.
2 Level-2 paths.
VX-EAST#sho mint info
Mint ID: 12.40.AD.09
0 Level-1 neighbors
Level-1 LSP DB size 1 LSPs (1 KB)
1 Level-2 neighbors
Level-2 LSP DB size 2 LSPs (1 KB)
Level-2 gateway is self
Level-2 router
Reachable adopters: 12.40.AD.09,12.40.AD.08
Max level-1 path cost: 0 (to 12.40.AD.09)

Re: WiNG vx9000 not clustering since being relocated to new HyperV host

Eric_Burke — Tue, 10 Mar 2020 03:10:42 GMT

@Chris Kelly , the AP’s do have IP’s in that VLAN, but as I understand it, they’re not really necessary. They get a DHCP address in the same VLAN that the controller has a subinterface. The vx9000 has a single GigE trunk, untagged into our management vlan and tagged into the adption vlan. The controllers can reach one another over the management network via their local default gateway (the firewall).

Re: WiNG vx9000 not clustering since being relocated to new HyperV host

ckelly — Tue, 10 Mar 2020 03:30:22 GMT

Eric,

There don’t appear to be any APs currently adopted. Is that correct? (you can check by running: show adoption status)

I do see on both controllers that they each have one Level-2 neighbor...and I’m guessing that that neighbor is the other controller.

As for the APs and IP addresses - If the APs do have layer-3 access for adopting on their management VLAN, that’s good. But based on your comment about the trunk on the controller and a tagged VLAN “into the adoption VLAN”...I’m not sure about this.

Essentially, what I would want to see is the APs using layer-3 (IP address) on whatever VLAN is needed to be able to adopt to the controllers.

(it may come down, if the culprit isn’t found fairly soon, to taking a look at the full configs from each controller. Sometimes a misconfiguration in a place that I”m assuming is correct can be hiding out somewhere and cause you to end up chasing your tail)

Again, for now though...fix the controller cluster members entries area but you need to use the correct MINT routing level. If in doubt, use Level-2. But this would also mean that you need to ensure that the APs are also adopting using MINT Level-2.

Re: WiNG vx9000 not clustering since being relocated to new HyperV host

Matt_Unger — Tue, 10 Mar 2020 03:35:52 GMT

HI Chris,

Just to confirm all of the AP’s are currently offline. They are being installed at the new customer site.

Re: WiNG vx9000 not clustering since being relocated to new HyperV host

ckelly — Tue, 10 Mar 2020 07:40:12 GMT

Eric,

We recently deployed a pair of vx9000’s in HyperV. The controllers are each on a different host. When we started building, we only had one host ready so we built the pair on that box. Now that both hosts are up, we exported and imported one of the units, Same IP address, just a new host. We found that mistakenly we didn’t clone the MAC and lost the licensing. We then set the MAC back, and confirmed that the vx9000’s coudl still ping each other.

Can you confirm that from those original 2 VX9000’s that were built that only 1 of them is still running?

Re: WiNG vx9000 not clustering since being relocated to new HyperV host

Matt_Unger — Tue, 10 Mar 2020 08:59:03 GMT

Chris,

Both controllers are currently up and running. West is the one that has not been modified, East is the one that was built on one server and then migrated to a new physical server.

Re: WiNG vx9000 not clustering since being relocated to new HyperV host

ckelly — Tue, 10 Mar 2020 09:21:40 GMT

Understood. In that case, just ensure that the original EAST controller is shut down.

Re: WiNG vx9000 not clustering since being relocated to new HyperV host

Matt_Unger — Tue, 10 Mar 2020 09:28:17 GMT

Original East VM is definitely offline.

Re: WiNG vx9000 not clustering since being relocated to new HyperV host

Matt_Unger — Tue, 10 Mar 2020 22:30:59 GMT

@Chris Kelly I removed the Mint Level 1 entries and now both controllers are able to see each other using Mint Level 2. Thank you for you help with this, it is very much appreciated!