WiNG vx9000 not clustering since being relocated to new HyperV host

[Eric_Burke]

We recently deployed a pair of vx9000’s in HyperV. The controllers are each on a different host. When we started building, we only had one host ready so we built the pair on that box. Now that both hosts are up, we exported and imported one of the units,  Same IP address, just a new host. We found that mistakenly we didn’t clone the MAC and lost the licensing. We then set the MAC back, and confirmed that the vx9000’s coudl still ping eachother. They’re sending hello’s via MINT. A support engineer helped us to realize that the license got dropped, presumably when getting the wrong MAC, so we re-licensed it. Still, we could not get the cluster back up. The engineer dropped the node we’d moved and manually rejoined the cluster. This seemed to work for a brief couple of minutes, but then we kept seeing the active role shifting back and forth between controllers. On the relocated unit, we’re seeing the following message:

CMASTER_CFG_UPDATE_FAIL: Cluster Master Rejecting Config from unknown host. Not sure what else we need to do to resolve the issue?  Any guidance?

[ckelly]

Matt/Eric,

Looking at the GUI entries on the controller, you have the same IP address listed as using both MINT level-1 and level-2 (10.200.247.192).  This is not correct and will certainly cause issues.  You should only have a singe MiNT routing level defined here for both entries for cluster members.  This MINT level should match the MINT level that your APs are using for adoption (this is important!)

Which controller is 10.200.247.192?  (East or West?)

Which controller is 10.66.64.192?  (East or West?)

From the CLI queries, things appear normal except that from the West controller’s perspective, the standby controller (East is down).  This would normally be due to a loss of MINT connectivity.

From the East controller’s perspective, it’s undergoing an election process.

It boils down to the controller’s not properly communicating with each other and each seems to be thinking that the other controller is down...

I haven’t had time to sift through the debug output yet (formatting was lost and makes it difficult to read).

For now, you need to get the cluster member setup corrected (with correct IP addresses and MINT level)

[Matt_Unger]

I am working with Eric on this issue and would like to let you know what I am seeing on the web interface for these controllers.

From the West:

From the East

[Eric_Burke]

Here’s the debug from the “primary” controller:

VX-WEST*>en VX-WEST*#debug cfgd cluster VX-WEST*#logging monitor debugging VX-WEST*#Mar 09 19:36:48 2020: USER: main.pyo: received packet START_ELECTION ver 1 prio 128 master period 0 rank 220 dc 1 id 12.40.AD.09 Mar 09 19:36:48 2020: USER: main.pyo: received packet from unknown member START_ELECTION ver 1 prio 128 master period 0 rank 220 dc 1 id 12.40.AD.09 Mar 09 19:36:48 2020: USER: main.pyo: received packet CANDIDATE ver 1 prio 128 master period 0 rank 220 dc 1 id 12.40.AD.09 Mar 09 19:36:48 2020: USER: main.pyo: received packet from unknown member CANDIDATE ver 1 prio 128 master period 0 rank 220 dc 1 id 12.40.AD.09 Mar 09 19:36:49 2020: USER: main.pyo: election recv timeout Mar 09 19:36:49 2020: USER: main.pyo: completed election master is 12.40.AD.08 Mar 09 19:36:49 2020: USER: main.pyo: new member 12.40.AD.08 Mar 09 19:36:49 2020: USER: main.pyo: adding device 12.40.AD.08 [00-15-5D-40-AD-08] as existing cluster member Mar 09 19:36:49 2020: USER: main.pyo: check if cluster state transition is required, mcount 1 member_id 00.00.00.00 member_changed_state 0 Mar 09 19:36:49 2020: USER: main.pyo: active cluster member is down; standby need to become active Mar 09 19:36:49 2020: USER: main.pyo: publish own cluster status=[active] internally Mar 09 19:36:49 2020: VX-WEST : %DIAG-6-NEW_LED_STATE: LED state message LED_ACTIVE_ADOPTING from module CFGD Mar 09 19:36:52 2020: USER: main.pyo: publish own cluster status=[active] internally Mar 09 19:36:52 2020: USER: main.pyo: handle_cluster_member_update members: ['12.40.AD.08', '12.40.AD.09'] Mar 09 19:36:52 2020: USER: main.pyo: handle_cluster_member_update new members, start pending election tasklet cluster_member_update_count=796 Mar 09 19:36:52 2020: USER: main.pyo: pending_election_tasklet(): cluster_member_update_count=796 member_update_count=796 members=['12.40.AD.08', '12.40.AD.09'] Mar 09 19:37:02 2020: USER: main.pyo: pending_election_tasklet(): after hysteresis delay cluster_member_update_count=796 member_update_count=796 Mar 09 19:37:02 2020: USER: main.pyo: start_election(): the_election.candidate_members: ['12.40.AD.08'] members: ['12.40.AD.08', '12.40.AD.09'] Mar 09 19:37:02 2020: USER: main.pyo: updating new members Mar 09 19:37:02 2020: USER: main.pyo: Updating new members ['12.40.AD.09'] Mar 09 19:37:02 2020: USER: main.pyo: created new local pkt: MASTER_UPDATE ver 1 prio 255 master period 57551 rank 220 dc 0 id 12.40.AD.08 Mar 09 19:37:02 2020: USER: main.pyo: dequeue - need to run election Mar 09 19:37:02 2020: USER: main.pyo: starting election forced=False immediate_master_election=False Mar 09 19:37:02 2020: USER: main.pyo: created new local pkt: CANDIDATE ver 1 prio 255 master period 57551 rank 220 dc 1 id 12.40.AD.08 Mar 09 19:37:02 2020: USER: main.pyo: created new local pkt: START_ELECTION ver 1 prio 255 master period 57551 rank 220 dc 1 id 12.40.AD.08 Mar 09 19:37:02 2020: USER: main.pyo: broadcasting election start Mar 09 19:37:02 2020: USER: main.pyo: announcing self as candidate Mar 09 19:37:02 2020: USER: main.pyo: cluster reset() member count 1 connected member count 1 Mar 09 19:37:02 2020: USER: main.pyo: adding device 12.40.AD.08 [00-15-5D-40-AD-08] as new cluster member Mar 09 19:37:02 2020: USER: main.pyo: canceling ask_msg_update_timer Mar 09 19:37:02 2020: VX-WEST : %DIAG-6-NEW_LED_STATE: LED state message LED_ACTIVE_ADOPTING from module CFGD Mar 09 19:37:02 2020: USER: main.pyo: received packet CANDIDATE ver 1 prio 128 master period 0 rank 220 dc 1 id 12.40.AD.09 Mar 09 19:37:02 2020: USER: main.pyo: Local election score improved, 2 known configs Mar 09 19:37:02 2020: USER: main.pyo: created new local pkt: CANDIDATE ver 1 prio 255 master period 57551 rank 220 dc 2 id 12.40.AD.08 Mar 09 19:37:02 2020: USER: main.pyo: best set to 12.40.AD.08 (CANDIDATE ver 1 prio 255 master period 57551 rank 220 dc 2 id 12.40.AD.08) Mar 09 19:37:02 2020: USER: main.pyo: retaining old candidate 12.40.AD.08 as best, not 12.40.AD.09 Mar 09 19:37:02 2020: USER: main.pyo: received packet CANDIDATE ver 1 prio 128 master period 0 rank 220 dc 2 id 12.40.AD.09 Mar 09 19:37:02 2020: USER: main.pyo: retaining old candidate 12.40.AD.08 as best, not 12.40.AD.09 Mar 09 19:37:17 2020: USER: main.pyo: election recv timeout Mar 09 19:37:17 2020: USER: main.pyo: Announcing self as master Mar 09 19:37:17 2020: USER: main.pyo: created new local pkt: ELECTION_WINNER ver 1 prio 255 master period 57566 rank 220 dc 2 id 12.40.AD.08 Mar 09 19:37:18 2020: VX-WEST : %SYSTEM-5-UI_USER_AUTH_SUCCESS: UI user 'admin' from: '10.66.64.225' authentication successful Mar 09 19:37:20 2020: USER: main.pyo: ClusterGenerator: klist=['00-15-5D-40-AD-08', '00-15-5D-40-AD-09'] Mar 09 19:37:31 2020: USER: main.pyo: got cluster msg 3 from 12.40.AD.09 (306228489, 0, ('i', 24576, '10.200.247.192')) Mar 09 19:37:31 2020: USER: main.pyo: Received member update msg 3 from unknown member 12.40.AD.09 (306228489, 0, ('i', 24576, '10.200.247.192')) Mar 09 19:37:31 2020: USER: main.pyo: got cluster msg 3 from 12.40.AD.09 (306228489, 0, ('i', 24576, '10.200.247.192')) Mar 09 19:37:31 2020: USER: main.pyo: Received member update msg 3 from unknown member 12.40.AD.09 (306228489, 0, ('i', 24576, '10.200.247.192')) Mar 09 19:37:31 2020: VX-EAST : %DIAG-6-NEW_LED_STATE: LED state message LED_ACTIVE_ADOPTING from module CFGD Mar 09 19:37:32 2020: USER: main.pyo: election recv timeout Mar 09 19:37:32 2020: USER: main.pyo: completed election master is 12.40.AD.08 Mar 09 19:37:32 2020: USER: main.pyo: new member 12.40.AD.08 Mar 09 19:37:32 2020: USER: main.pyo: adding device 12.40.AD.08 [00-15-5D-40-AD-08] as existing cluster member Mar 09 19:37:32 2020: USER: main.pyo: new member 12.40.AD.09 Mar 09 19:37:32 2020: USER: main.pyo: adding device 12.40.AD.09 [00-15-5D-40-AD-09] as new cluster member Mar 09 19:37:32 2020: USER: main.pyo: mint_id = 12.40.AD.09 APLIC_INSTALLED=0 APLIC_ACQUIRED=0 APLIC_GRANTED=0 APLIC_PENDING=0 APLIC_TOTAL=0 Mar 09 19:37:32 2020: USER: main.pyo: Mar 09 19:37:32 2020: USER: main.pyo: mint_id = 12.40.AD.09 AAPLIC_INSTALLED=64 AAPLIC_ACQUIRED=0 AAPLIC_GRANTED=0 AAPLIC_PENDING=0 AAPLIC_TOTAL=64 Mar 09 19:37:32 2020: USER: main.pyo: Mar 09 19:37:32 2020: USER: main.pyo: add_member() send update message to member 12.40.AD.09 Mar 09 19:37:32 2020: USER: main.pyo: send update req message to member 12.40.AD.09 Mar 09 19:37:32 2020: USER: main.pyo: master-updater woke up - 1 changes in the list Mar 09 19:37:32 2020: USER: main.pyo: tx_update CS_ACT_CFG_FULL - revision 3 Mar 09 19:37:32 2020: USER: main.pyo: tx_update member rev is 0 Mar 09 19:37:32 2020: USER: main.pyo: tx-update send update raw to 12.40.AD.09 Mar 09 19:37:32 2020: USER: main.pyo: sending UPDATE revision: 3 hdr size: 20, data size: 50539 Mar 09 19:37:32 2020: USER: main.pyo: CS_Send() connecting to 12.40.AD.09 Mar 09 19:37:32 2020: VX-WEST : %DIAG-6-NEW_LED_STATE: LED state message LED_ACTIVE_ADOPTING from module CFGD Mar 09 19:37:32 2020: USER: main.pyo: CS_Send() connected Mar 09 19:37:33 2020: VX-EAST : %CFGD-6-ACL_RULE_ALTERED: USER: cfgd session 13: ACL BROADCAST-MULTICAST-CONTROL rule is getting altered Mar 09 19:37:33 2020: VX-EAST : %CFGD-6-ACL_RULE_ALTERED: USER: cfgd session 13: ACL PERMIT-ARP-AND-IPv4 rule is getting altered Mar 09 19:37:33 2020: VX-EAST : %SYSTEM-6-CONFIG_COMMIT: Configuration commit by user 'cfgd' (update own config) from '127.0.0.1' Mar 09 19:37:33 2020: USER: main.pyo: cluster config update OK, current revision: 3, expected revision: 3 Mar 09 19:37:33 2020: USER: main.pyo: tx_update CS_ACT_CFG_FULL - revision 3 host 12.40.AD.09 is OK Mar 09 19:37:33 2020: VX-EAST : %SYSTEM-6-CONFIG_REVISION: Configuration revision updated to 3 from 3 Mar 09 19:37:58 2020: VX-WEST : %SYSTEM-5-UI_USER_AUTH_SUCCESS: UI user 'admin' from: '10.66.64.225' authentication successful Mar 09 19:38:01 2020: USER: main.pyo: ClusterGenerator: klist=['00-15-5D-40-AD-08', '00-15-5D-40-AD-09'] Mar 09 19:38:04 2020: USER: main.pyo: publish own cluster status=[active] internally Mar 09 19:38:04 2020: USER: main.pyo: handle_cluster_member_update members: ['12.40.AD.08'] Mar 09 19:38:04 2020: USER: main.pyo: handle_cluster_member_update fewer members cluster_member_update_count=797 Mar 09 19:38:04 2020: USER: main.pyo: start_election(): the_election.candidate_members: ['12.40.AD.08', '12.40.AD.09'] members: ['12.40.AD.08'] Mar 09 19:38:04 2020: USER: main.pyo: no change in master, member list changed Mar 09 19:38:04 2020: USER: main.pyo: dequeue - need to run election Mar 09 19:38:04 2020: USER: main.pyo: starting election forced=False immediate_master_election=False Mar 09 19:38:04 2020: USER: main.pyo: created new local pkt: CANDIDATE ver 1 prio 255 master period 57614 rank 220 dc 1 id 12.40.AD.08 Mar 09 19:38:04 2020: USER: main.pyo: created new local pkt: START_ELECTION ver 1 prio 255 master period 57614 rank 220 dc 1 id 12.40.AD.08 Mar 09 19:38:04 2020: USER: main.pyo: broadcasting election start Mar 09 19:38:04 2020: USER: main.pyo: announcing self as candidate Mar 09 19:38:04 2020: USER: main.pyo: cluster reset() member count 2 connected member count 2 Mar 09 19:38:04 2020: USER: main.pyo: adding device 12.40.AD.08 [00-15-5D-40-AD-08] as new cluster member Mar 09 19:38:04 2020: USER: main.pyo: canceling ask_msg_update_timer Mar 09 19:38:04 2020: VX-WEST : %DIAG-6-NEW_LED_STATE: LED state message LED_ACTIVE_ADOPTING from module CFGD Mar 09 19:38:19 2020: USER: main.pyo: election recv timeout Mar 09 19:38:19 2020: USER: main.pyo: Announcing self as master Mar 09 19:38:19 2020: USER: main.pyo: created new local pkt: ELECTION_WINNER ver 1 prio 255 master period 57629 rank 220 dc 1 id 12.40.AD.08 Mar 09 19:38:34 2020: VX-WEST : %DIAG-6-NEW_LED_STATE: LED state message LED_ACTIVE_ADOPTING from module CFGD Mar 09 19:38:34 2020: USER: main.pyo: election recv timeout Mar 09 19:38:34 2020: USER: main.pyo: completed election master is 12.40.AD.08 Mar 09 19:38:34 2020: USER: main.pyo: new member 12.40.AD.08 Mar 09 19:38:34 2020: USER: main.pyo: adding device 12.40.AD.08 [00-15-5D-40-AD-08] as existing cluster member Mar 09 19:38:34 2020: USER: main.pyo: check if cluster state transition is required, mcount 1 member_id 00.00.00.00 member_changed_state 0 Mar 09 19:38:34 2020: USER: main.pyo: active cluster member is down; standby need to become active Mar 09 19:38:34 2020: USER: main.pyo: publish own cluster status=[active] internally Mar 09 19:38:54 2020: USER: main.pyo: publish own cluster status=[active] internally Mar 09 19:38:54 2020: USER: main.pyo: handle_cluster_member_update members: ['12.40.AD.08', '12.40.AD.09'] Mar 09 19:38:54 2020: USER: main.pyo: handle_cluster_member_update new members, start pending election tasklet cluster_member_update_count=798 Mar 09 19:38:54 2020: USER: main.pyo: pending_election_tasklet(): cluster_member_update_count=798 member_update_count=798 members=['12.40.AD.08', '12.40.AD.09'] Mar 09 19:38:56 2020: USER: main.pyo: publish own cluster status=[active] internally Mar 09 19:38:56 2020: USER: main.pyo: handle_cluster_member_update members: ['12.40.AD.08'] Mar 09 19:38:56 2020: USER: main.pyo: handle_cluster_member_update fewer members cluster_member_update_count=799 Mar 09 19:39:04 2020: USER: main.pyo: Peer 12.40.AD.09 connected Mar 09 19:39:04 2020: USER: main.pyo: received pkt MASTER_UPDATE ver 1 prio 128 master period 30 rank 220 dc 0 id 12.40.AD.09 Mar 09 19:39:04 2020: USER: main.pyo: Received packet from unknown member MASTER_UPDATE ver 1 prio 128 master period 30 rank 220 dc 0 id 12.40.AD.09 Mar 09 19:39:04 2020: USER: main.pyo: received pkt START_ELECTION ver 1 prio 128 master period 30 rank 220 dc 1 id 12.40.AD.09 Mar 09 19:39:04 2020: USER: main.pyo: Received packet from unknown member START_ELECTION ver 1 prio 128 master period 30 rank 220 dc 1 id 12.40.AD.09 Mar 09 19:39:04 2020: USER: main.pyo: received pkt CANDIDATE ver 1 prio 128 master period 30 rank 220 dc 1 id 12.40.AD.09 Mar 09 19:39:04 2020: USER: main.pyo: Received packet from unknown member CANDIDATE ver 1 prio 128 master period 30 rank 220 dc 1 id 12.40.AD.09 Mar 09 19:39:04 2020: USER: main.pyo: pending_election_tasklet(): after hysteresis delay cluster_member_update_count=799 member_update_count=798 Mar 09 19:39:04 2020: USER: main.pyo: do not start election due to member list change: cluster_member_update_count=799 member_update_count=798 Mar 09 19:39:09 2020: USER: main.pyo: Ignore Exception Mar 09 19:39:19 2020: USER: main.pyo: received pkt START_ELECTION ver 1 prio 128 master period 45 rank 220 dc 1 id 12.40.AD.09 Mar 09 19:39:19 2020: USER: main.pyo: Received packet from unknown member START_ELECTION ver 1 prio 128 master period 45 rank 220 dc 1 id 12.40.AD.09 Mar 09 19:39:19 2020: USER: main.pyo: received pkt CANDIDATE ver 1 prio 128 master period 45 rank 220 dc 1 id 12.40.AD.09 Mar 09 19:39:19 2020: USER: main.pyo: Received packet from unknown member CANDIDATE ver 1 prio 128 master period 45 rank 220 dc 1 id 12.40.AD.09 Mar 09 19:39:34 2020: USER: main.pyo: received pkt START_ELECTION ver 1 prio 128 master period 60 rank 220 dc 1 id 12.40.AD.09 Mar 09 19:39:34 2020: USER: main.pyo: Received packet from unknown member START_ELECTION ver 1 prio 128 master period 60 rank 220 dc 1 id 12.40.AD.09 Mar 09 19:39:34 2020: USER: main.pyo: received pkt CANDIDATE ver 1 prio 128 master period 60 rank 220 dc 1 id 12.40.AD.09 Mar 09 19:39:34 2020: USER: main.pyo: Received packet from unknown member CANDIDATE ver 1 prio 128 master period 60 rank 220 dc 1 id 12.40.AD.09 Mar 09 19:39:49 2020: USER: main.pyo: received pkt START_ELECTION ver 1 prio 128 master period 75 rank 220 dc 1 id 12.40.AD.09 Mar 09 19:39:49 2020: USER: main.pyo: Received packet from unknown member START_ELECTION ver 1 prio 128 master period 75 rank 220 dc 1 id 12.40.AD.09 Mar 09 19:39:49 2020: USER: main.pyo: received pkt CANDIDATE ver 1 prio 128 master period 75 rank 220 dc 1 id 12.40.AD.09 Mar 09 19:39:49 2020: USER: main.pyo: Received packet from unknown member CANDIDATE ver 1 prio 128 master period 75 rank 220 dc 1 id 12.40.AD.09 exit 

[Eric_Burke]

Here is the WEST controller (the one that stayed up, is primary, never moved):

VX-WEST*#show cluster status Cluster Runtime Information Protocol version : 1 Cluster operational state : active AP license : 0 AAP license : 128 AP count : 0 AAP count : 0 Max AP adoption capacity : 25600 Number of connected member(s): 0 VX-WEST*# VX-WEST*#show cluster configuration Cluster Configuration Information Name : HeathVillage Configured Mode : Active Master Priority : 255 Force configured state : Disabled Force configured state delay : 5 minutes Handle STP : Disabled Radius Counter DB Sync Time : 5 minutes VX-WEST*# VX-WEST*#show cluster members ------------------------------------------------------------------------------------------ HOSTNAME MEMBER-ID MAC MASTER OPERATIONAL-STATE LAST-SEEN ------------------------------------------------------------------------------------------ VX-WEST* 12.40.AD.08 00-15-5D-40-AD-08 True active self VX-EAST 00-15-5D-40-AD-09 False down ------------------------------------------------------------------------------------------ VX-WEST*# VX-WEST*#show cluster history ----------------------------------------------------------------------------------------------------------------------------- HOSTNAME MEMBER-ID EVENT TIME-STAMP REASON ----------------------------------------------------------------------------------------------------------------------------- VX-WEST 12.40.AD.08 start-election 2020-03-09 19:31:34 Member 12.40.AD.09 joined VX-WEST 12.40.AD.08 is-active 2020-03-09 18:20:32 Primary down, currently active, stay active VX-WEST 12.40.AD.08 start-election 2020-03-09 18:20:02 Member 12.40.AD.09 down VX-WEST 12.40.AD.08 start-config-sync 2020-03-09 18:19:11 Member 12.40.AD.08 won election and is the cluster master VX-WEST 12.40.AD.08 start-election 2020-03-09 18:18:36 Member 12.40.AD.09 joined VX-WEST 12.40.AD.08 is-active 2020-03-09 18:18:14 Primary down, currently active, stay active VX-WEST 12.40.AD.08 start-election 2020-03-09 18:17:44 Member 12.40.AD.09 down VX-WEST 12.40.AD.08 is-active 2020-03-09 18:16:51 No configured standby members, stay active VX-WEST 12.40.AD.08 handle-member-update 2020-03-09 18:16:36 Member 12.40.AD.09 state change to active VX-WEST 12.40.AD.08 config-update-success 2020-03-09 18:14:43 Successfully applied config VX-WEST 12.40.AD.08 is-active 2020-03-09 18:14:36 Primary down, currently active, stay active VX-WEST 12.40.AD.08 config-update-rejected 2020-03-09 18:14:36 ['rejecting config from unknown host'] VX-WEST 12.40.AD.08 start-config-sync 2020-03-09 18:14:36 Member 12.40.AD.08 won election and is the cluster master VX-WEST 12.40.AD.08 start-election 2020-03-09 18:14:06 Member 12.40.AD.09 joined VX-WEST 12.40.AD.08 is-active 2020-03-09 18:10:49 Primary down, currently active, stay active VX-WEST 12.40.AD.08 start-election 2020-03-09 18:10:19 Member 12.40.AD.09 down VX-WEST 12.40.AD.08 is-active 2020-03-09 18:07:21 No configured standby members, stay active VX-WEST 12.40.AD.08 config-update-success 2020-03-09 18:07:06 Successfully applied config VX-WEST 12.40.AD.08 handle-member-update 2020-03-09 18:07:06 Member 12.40.AD.09 state change to active VX-WEST 12.40.AD.08 is-active 2020-03-09 18:07:06 Primary down, currently active, stay active VX-WEST 12.40.AD.08 start-config-sync 2020-03-09 18:07:06 Member 12.40.AD.08 won election and is the cluster master VX-WEST 12.40.AD.08 start-election 2020-03-09 18:06:35 Member 12.40.AD.09 joined VX-WEST 12.40.AD.08 is-active 2020-03-09 18:05:11 Primary down, currently active, stay active VX-WEST 12.40.AD.08 start-election 2020-03-09 18:03:59 Member 12.40.AD.09 joined VX-WEST 12.40.AD.08 is-active 2020-03-09 18:01:52 Primary down, currently active, stay active VX-WEST 12.40.AD.08 start-election 2020-03-09 18:01:22 Member 12.40.AD.09 down VX-WEST 12.40.AD.08 is-active 2020-03-09 18:00:21 No configured standby members, stay active VX-WEST 12.40.AD.08 config-update-success 2020-03-09 18:00:13 Successfully applied config VX-WEST 12.40.AD.08 handle-member-update 2020-03-09 18:00:06 Member 12.40.AD.09 state change to active VX-WEST 12.40.AD.08 config-update-rejected 2020-03-09 18:00:05 ['rejecting config from unknown host'] VX-WEST 12.40.AD.08 is-active 2020-03-09 18:00:05 Primary down, currently active, stay active VX-WEST 12.40.AD.08 start-config-sync 2020-03-09 18:00:05 Member 12.40.AD.08 won election and is the cluster master VX-WEST 12.40.AD.08 start-election 2020-03-09 17:59:35 Member 12.40.AD.09 joined VX-WEST 12.40.AD.08 is-active 2020-03-09 17:59:19 Primary down, currently active, stay active VX-WEST 12.40.AD.08 start-election 2020-03-09 17:58:49 Member 12.40.AD.09 down VX-WEST 12.40.AD.08 is-active 2020-03-09 17:58:21 No configured standby members, stay active VX-WEST 12.40.AD.08 config-update-success 2020-03-09 17:58:13 Successfully applied config VX-WEST 12.40.AD.08 handle-member-update 2020-03-09 17:58:06 Member 12.40.AD.09 state change to active VX-WEST 12.40.AD.08 is-active 2020-03-09 17:58:06 Primary down, currently active, stay active VX-WEST 12.40.AD.08 config-update-rejected 2020-03-09 17:58:05 ['rejecting config from unknown host'] VX-WEST 12.40.AD.08 start-config-sync 2020-03-09 17:58:05 Member 12.40.AD.08 won election and is the cluster master VX-WEST 12.40.AD.08 start-election 2020-03-09 17:57:35 Member 12.40.AD.09 joined VX-WEST 12.40.AD.08 is-active 2020-03-09 17:56:58 Primary down, currently active, stay active VX-WEST 12.40.AD.08 start-election 2020-03-09 17:55:44 Member 12.40.AD.09 down VX-WEST 12.40.AD.08 config-update-success 2020-03-09 17:54:36 Successfully applied config VX-WEST 12.40.AD.08 is-active 2020-03-09 17:54:35 Primary down, currently active, stay active ----------------------------------------------------------------------------------------------------------------------------- 

Here is the one we moved, rejoined to the cluster after fixing the license issue and setting the MAC back to the original address:

VX-EAST>show cluster status Cluster Runtime Information Protocol version : 1 Cluster operational state : standby AP license : 0 AAP license : 128 AP count : 0 AAP count : 0 Max AP adoption capacity : 25600 Number of connected member(s): 0 VX-EAST> VX-EAST>show cluster configuration Cluster Configuration Information Name : HeathVillage Configured Mode : Active Master Priority : 128 Force configured state : Disabled Force configured state delay : 5 minutes Handle STP : Disabled Radius Counter DB Sync Time : 5 minutes VX-EAST> VX-EAST>show cluster members Cluster master election in progress Configured cluster members 00-15-5D-40-AD-08 00-15-5D-40-AD-09 VX-EAST> VX-EAST>show cluster history ----------------------------------------------------------------------------------------------------------------------------- HOSTNAME MEMBER-ID EVENT TIME-STAMP REASON ----------------------------------------------------------------------------------------------------------------------------- VX-EAST 12.40.AD.09 start-election 2020-03-09 19:02:09 Member 12.40.AD.08 joined VX-EAST 12.40.AD.09 is-active 2020-03-09 19:01:41 Primary down, currently active, stay active VX-EAST 12.40.AD.09 start-election 2020-03-09 19:01:11 Member 12.40.AD.08 down VX-EAST 12.40.AD.09 config-update-success 2020-03-09 18:59:12 Successfully applied config VX-EAST 12.40.AD.09 is-active 2020-03-09 18:57:39 Primary down, currently standby, become active VX-EAST 12.40.AD.09 start-election 2020-03-09 18:56:11 Member 12.40.AD.09 joined VX-EAST 12.40.AD.09 start-election 2020-03-09 17:28:23 Member 12.40.AD.08 down VX-EAST 12.40.AD.09 config-update-success 2020-03-09 17:25:24 Successfully applied config VX-EAST 12.40.AD.09 start-election 2020-03-09 17:23:50 Member 12.40.AD.08 joined VX-EAST 12.40.AD.09 is-active 2020-03-09 17:23:04 Primary down, currently active, stay active VX-EAST 12.40.AD.09 start-election 2020-03-09 17:21:50 Member 12.40.AD.08 joined VX-EAST 12.40.AD.09 is-active 2020-03-09 17:21:33 Primary down, currently active, stay active VX-EAST 12.40.AD.09 start-election 2020-03-09 17:21:03 Member 12.40.AD.08 down VX-EAST 12.40.AD.09 config-update-success 2020-03-09 17:17:53 Successfully applied config VX-EAST 12.40.AD.09 start-election 2020-03-09 17:16:35 Member 12.40.AD.08 joined VX-EAST 12.40.AD.09 is-active 2020-03-09 17:15:59 Primary down, currently active, stay active VX-EAST 12.40.AD.09 start-election 2020-03-09 17:14:20 Member 12.40.AD.08 joined VX-EAST 12.40.AD.09 is-active 2020-03-09 17:12:43 Primary down, currently active, stay active VX-EAST 12.40.AD.09 start-election 2020-03-09 17:12:13 Member 12.40.AD.08 down VX-EAST 12.40.AD.09 config-update-success 2020-03-09 17:11:05 Successfully applied config VX-EAST 12.40.AD.09 start-election 2020-03-09 17:09:41 Member 12.40.AD.08 down VX-EAST 12.40.AD.09 config-update-success 2020-03-09 17:09:06 Successfully applied config VX-EAST 12.40.AD.09 start-election 2020-03-09 17:06:39 Member 12.40.AD.08 down VX-EAST 12.40.AD.09 config-update-success 2020-03-09 17:05:32 Successfully applied config VX-EAST 12.40.AD.09 start-election 2020-03-09 17:03:36 Member 12.40.AD.08 joined VX-EAST 12.40.AD.09 is-active 2020-03-09 17:03:06 Primary down, currently active, stay active VX-EAST 12.40.AD.09 start-election 2020-03-09 17:02:36 Member 12.40.AD.08 down VX-EAST 12.40.AD.09 config-update-success 2020-03-09 17:00:18 Successfully applied config VX-EAST 12.40.AD.09 start-election 2020-03-09 16:59:06 Member 12.40.AD.08 joined VX-EAST 12.40.AD.09 is-active 2020-03-09 16:59:04 Primary down, currently active, stay active VX-EAST 12.40.AD.09 start-election 2020-03-09 16:57:51 Member 12.40.AD.08 joined VX-EAST 12.40.AD.09 is-active 2020-03-09 16:57:48 Primary down, currently active, stay active VX-EAST 12.40.AD.09 start-election 2020-03-09 16:54:31 Member 12.40.AD.08 down VX-EAST 12.40.AD.09 config-update-success 2020-03-09 16:52:48 Successfully applied config VX-EAST 12.40.AD.09 start-election 2020-03-09 16:50:43 Member 12.40.AD.08 down VX-EAST 12.40.AD.09 start-election 2020-03-09 16:47:37 Member 12.40.AD.08 joined VX-EAST 12.40.AD.09 is-active 2020-03-09 16:47:26 Primary down, currently active, stay active VX-EAST 12.40.AD.09 start-election 2020-03-09 16:45:25 Member 12.40.AD.08 down VX-EAST 12.40.AD.09 config-update-success 2020-03-09 16:42:33 Successfully applied config VX-EAST 12.40.AD.09 start-election 2020-03-09 16:41:53 Member 12.40.AD.08 joined VX-EAST 12.40.AD.09 is-active 2020-03-09 16:37:49 Primary down, currently active, stay active VX-EAST 12.40.AD.09 start-election 2020-03-09 16:37:19 Member 12.40.AD.08 down VX-EAST 12.40.AD.09 config-update-success 2020-03-09 16:36:29 Successfully applied config VX-EAST 12.40.AD.09 start-election 2020-03-09 16:34:23 Member 12.40.AD.08 joined VX-EAST 12.40.AD.09 is-active 2020-03-09 16:34:17 Primary down, currently active, stay active VX-EAST 12.40.AD.09 start-election 2020-03-09 16:32:24 Member 12.40.AD.08 joined VX-EAST 12.40.AD.09 is-active 2020-03-09 16:32:01 Primary down, currently active, stay active VX-EAST 12.40.AD.09 start-election 2020-03-09 16:31:31 Member 12.40.AD.08 down VX-EAST 12.40.AD.09 config-update-success 2020-03-09 16:31:25 Successfully applied config VX-EAST 12.40.AD.09 start-election 2020-03-09 16:30:23 Member 12.40.AD.08 joined -----------------------------------------------------------------------------------------------------------------------------

Debug coming in next...

[ckelly]

Eric,

Can you give the output for the following:

(On BOTH cluster member, run these)

show cluster status

show cluster configuration

show cluster members

show cluster history

Then, on the Primary cluster member, let’s try some debugging: (let run long enough to see cluster related messages)  The output should hopefully reveal a little more of what’s happening here.

# debug cfgd cluster

# logging monitor debugging