WiNG vx9000 not clustering since being relocated to new HyperV host

[Eric_Burke]

We recently deployed a pair of vx9000’s in HyperV. The controllers are each on a different host. When we started building, we only had one host ready so we built the pair on that box. Now that both hosts are up, we exported and imported one of the units,  Same IP address, just a new host. We found that mistakenly we didn’t clone the MAC and lost the licensing. We then set the MAC back, and confirmed that the vx9000’s coudl still ping eachother. They’re sending hello’s via MINT. A support engineer helped us to realize that the license got dropped, presumably when getting the wrong MAC, so we re-licensed it. Still, we could not get the cluster back up. The engineer dropped the node we’d moved and manually rejoined the cluster. This seemed to work for a brief couple of minutes, but then we kept seeing the active role shifting back and forth between controllers. On the relocated unit, we’re seeing the following message:

CMASTER_CFG_UPDATE_FAIL: Cluster Master Rejecting Config from unknown host. Not sure what else we need to do to resolve the issue?  Any guidance?

[Eric_Burke]

West:  

VX-WEST*#sho mint stats
0 Level-1 neighbors
Level-1 LSP DB size 1 LSPs (1 KB)
Last Level-1 SPFs took 0.000s
Level-1 SPF (re)calculated 1638 times.
1 Level-1 paths.
1 Level-2 neighbors
Level-2 LSP DB size 1 LSPs (1 KB)
Last Level-2 SPFs took 0.000s
Level-2 SPF (re)calculated 9136 times.
1 Level-2 paths.
VX-WEST*#sho mint info
Mint ID: 12.40.AD.08
0 Level-1 neighbors
Level-1 LSP DB size 1 LSPs (1 KB)
1 Level-2 neighbors
Level-2 LSP DB size 1 LSPs (1 KB)
Level-2 gateway is unreachable
Reachable adopters: 12.40.AD.08
Max level-1 path cost: 0 (to 12.40.AD.08)

EAST:

VX-EAST#sho mint stats
0 Level-1 neighbors
Level-1 LSP DB size 1 LSPs (1 KB)
Last Level-1 SPFs took 0.000s
Level-1 SPF (re)calculated 31 times.
1 Level-1 paths.
1 Level-2 neighbors
Level-2 LSP DB size 2 LSPs (1 KB)
Last Level-2 SPFs took 0.000s
Level-2 SPF (re)calculated 61 times.
2 Level-2 paths.
VX-EAST#sho mint info
Mint ID: 12.40.AD.09
0 Level-1 neighbors
Level-1 LSP DB size 1 LSPs (1 KB)
1 Level-2 neighbors
Level-2 LSP DB size 2 LSPs (1 KB)
Level-2 gateway is self
Level-2 router
Reachable adopters: 12.40.AD.09,12.40.AD.08
Max level-1 path cost: 0 (to 12.40.AD.09)

[ckelly]

Eric,

To confirm, the APs are adopting to the controllers at layer-2?  The APs don’t have IP addresses on their management interfaces?

Controllers in a cluster being co-located is perfectly normal.  How you described the intended fail-over is a common setup and is normal operation.

As for MINT though, you need to ensure that however your APs are adopting (the MINT level they’re using) this also needs to be the MINT level that the controllers are setup to use for their clustering.

E.g., you don’t/can’t have APs adopting using level-1 MINT and then have your controllers clustered using MINT level-2.  They need to match.

[ckelly]

Matt,

Can you run this query:

# show mint stats

# show mint info

Your setup in the GUI should look like this...except I don’t know what your MINT routing level needs to be.  I’m hoping to determine this from the output of the above queries.

[Eric_Burke]

@Chris Kelly ,

We’re just getting into our first more “complex” deployment of WiNG here, so it may be an issue related to our intended design. Here’s the plan we discussed with pro services. The customer has 2 campuses, connected via fiber. In each campus, we’re running a HyperV host with a vx9000. The AP’s in each campus are adopted at layer 2 in their own, unique provisioning vlans. The controllers themselves are in separate L3 VLAN’s (as two different sites), connected by firewalls that have both point to point (over fiber) and VPN backup connections between them. Our understanding was that we could cluster the controllers at Layer 3 and should we lose a HyperV host, the AP’s connected to that associated controller could use the remote site’s controller as a backup. Is that incorrect? I think that means we have controllers with MINT2 links, but AP’s with site-specific MINT1...

[Matt_Unger]

10.66.64.192 is West

10.200.247.192 is East