topic Re: MAC move - up to 1000 times per hour! in ExtremeCloud IQ Controller

MAC move - up to 1000 times per hour!

Peer-JoachimK — Thu, 23 May 2024 13:48:09 GMT

Hi,

we had some hardware trouble a few days ago (POE disappeared on a X465P) together with a malicious telephone (broadcast storm). We detached the telephone (-> broadcast gone) and also replaced the switch. But since that time we see this kind of message on all stacks (using netlogin) in the network:

Slot-1: The authentication state of Network Login user XXX was cleared by policy
due to Admin Reset (MAC move), Mac XXX port 3:2 VLAN(s) "VL_XYZ" Protocol(s) "MAC"

We see 600-1000 MAC moves per hour. Some of the stack masters are rebooting after some time.
How is the event (Admin Rest) triggered ?
Is there a way to find some usefull information within the XMC ?

We did not touch the network configuration for the past 2 month.

Any help/idea is welcome!

Bye, Peer

Re: MAC move - up to 1000 times per hour!

Stefan_K_ — Thu, 23 May 2024 13:59:00 GMT

Hi @Peer-JoachimK

Depending on your network, 1000 MAC movements are not that much. Think about Roaming of Wifi Clients.

But besides that: It could be caused by a software defect (see https://extreme-networks.my.site.com/ExtrArticleDetail?an=000099551) or a loop.

You can try to find out from which port to which port the MACs are moving: https://extreme-networks.my.site.com/ExtrArticleDetail?an=000079713

To check for a loop, use ELRP:
https://extreme-networks.my.site.com/ExtrArticleDetail?an=000087084
https://extreme-networks.my.site.com/ExtrArticleDetail?an=000090973
https://extreme-networks.my.site.com/ExtrArticleDetail?an=000083207

Best regards
Stefan

Re: MAC move - up to 1000 times per hour!

Peer-JoachimK — Thu, 23 May 2024 14:35:27 GMT

Hi Stefan,

I'll have a look at the ELRP. I used the logging and could see the problem.
A PC attached to a switch is getting authenticated. (no wifi by the way 😉 ) - after a few minutes
the same MAC is shown at the uplink port and the ADMIN REST is caused.
We are using a fabric of 10 VSP 7400 as backbone, where all stacks are attached.
It would be interesting to delete the fdb entry in the fabric and see, if the VSPs are learning it correctly.
Bye, Peer

Re: MAC move - up to 1000 times per hour!

Stefan_K_ — Thu, 23 May 2024 15:33:54 GMT

@Peer-JoachimK wrote:
A PC attached to a switch is getting authenticated. (no wifi by the way 😉 ) - after a few minutes
the same MAC is shown at the uplink port and the ADMIN REST is caused.

Hi, this is indeed a typical symptom for a loop.

Best regards
Stefan

Re: MAC move - up to 1000 times per hour!

Peer-JoachimK — Mon, 27 May 2024 13:40:09 GMT

Hi,

I checked it on two stacks. No loops are detected....
From the physical topology it can not imagine how a loop should be created:

We have a fabric backbone (10 x VSP 7400) in a ring topology.
All stacks are attached to the ring using a SMLT upling.

The mac addresses are appearing at the uplink. So some VSP is announcing the
mac on different Uplink port. The local switch is doing a re authentication.

We see the changes in the fdb on the VSP for a short period, but we have no idea why.

Also some mac have more problem:

2357 B0:xx:yy...
1464 54:xx:yy...
702 B4:xx:yy...

Shouldn't a loop be impossible within a fabric ?

Bye, Peer

Re: MAC move - up to 1000 times per hour!

Peer-JoachimK — Tue, 28 May 2024 11:44:31 GMT

We fixed the problem, but still does not know what happened ....
One SMLT link to a stack caused the problem. Both VSP have been rebooted without success.
We shutdown one link and the problem was gone. The link is up and running now and we haven't seen any mac move.
The SMLT link must have been the problem. Why the reboot did not solve the problem is strange.

Bye, Peer