Strange LED blinking after upgrade

  • 0
  • 2
  • Question
  • Updated 3 years ago
  • Answered
Hi everybody!
I know, it sounds strange, but... I've recently upgrade my two MLAG-peers (X670V-48x) to 15.5.4.2 patch 1-5. So everething is good, switches work correct. Everybody is happy!

It was quite a shock when I saw swithes yesterday. All LEDs on both switches was synchronous blinking and they continue to do it until now. It looks like broadcast storm. Very strange because the network operate well.

Can anybody ssluggest me something?
Photo of eyeV

eyeV

  • 2,484 Points 2k badge 2x thumb

Posted 3 years ago

  • 0
  • 2
Photo of Drew C.

Drew C., Community Manager

  • 39,440 Points 20k badge 2x thumb
Hello friend, what version was installed previously?  I'm assuming you were happy with the rate in that previous version?
The LED blink rate was slowed under CR xos0059128, which is included in your patch version.
It's also in EXOS 16.1.1, EXOS 15.7.1+, and EXOS 15.6.2+.

Another note in the CR says that it's done for stacking... not sure if it was added to standalone switches.  Is yours a stack or standalone?
Photo of eyeV

eyeV

  • 2,484 Points 2k badge 2x thumb
Hi!

15.4.1.3-patch1-1 was installed previosly. I use two standalone switches whish is MLAG peers.

It's reaaly strange, but I have misterious fealing that it's not the XOS problem. Something happen in network.

to be continued...
Photo of Jed

Jed

  • 62 Points
Funny I just did this last night and I see the same crazy blinking and high cpu usage. Chunky response in the CLI and if you do a "show stacking" command forget about it you'll have to restart your telnet session. I shouldn't have touched anything I guess.
Photo of eyeV

eyeV

  • 2,484 Points 2k badge 2x thumb
My switches look great. CPU usage is low, everething is good. But LED blinking looks pretty creepy.
Photo of eyeV

eyeV

  • 2,484 Points 2k badge 2x thumb
So, I've done a lot of test. If no traffic in port it doesn't blink, if the traffic is in port it blinks in sync with the others.
Photo of Drew C.

Drew C., Community Manager

  • 39,302 Points 20k badge 2x thumb
Thanks for looking at this.  I think it would be a good idea to open a ticket with GTAC so that we can route this to engineering for further investigation.
Photo of Drew C.

Drew C., Community Manager

  • 39,440 Points 20k badge 2x thumb
I've had a chat with one of the Summit developers.  Here's what I learned:
The X670’s LED activity is updated by software, and not directly by the Broadcom chip like in other EXOS platforms.  This software process runs at a period of one second.  In each period, it checks if RX and TX counters have changed on each port.  Since the counter thread runs at a “slow” period, the LED processor is very likely to see link activity on every port (if any) at every scan.  So, it blinks the LEDs at the same time.  In summary, this is a hardware limitation on X670 that we are unfortunately unable to do anything about.
There were some changes to the timing and duration of the LED blinking, but it is still the same idea as before.  Previously, the scan ran a little faster but still blinked all LEDs at the same time on ports with activity.
Photo of eyeV

eyeV

  • 2,484 Points 2k badge 2x thumb
Thanks a lot Drew!

Hmm... I think I haven't noticed this blinking previously )