Strange LED blinking after upgrade


Userlevel 3
Hi everybody!
I know, it sounds strange, but... I've recently upgrade my two MLAG-peers (X670V-48x) to 15.5.4.2 patch 1-5. So everething is good, switches work correct. Everybody is happy!

It was quite a shock when I saw swithes yesterday. All LEDs on both switches was synchronous blinking and they continue to do it until now. It looks like broadcast storm. Very strange because the network operate well.

Can anybody ssluggest me something?

8 replies

Userlevel 7
Hello friend, what version was installed previously? I'm assuming you were happy with the rate in that previous version?
The LED blink rate was slowed under CR xos0059128, which is included in your patch version.
It's also in EXOS 16.1.1, EXOS 15.7.1+, and EXOS 15.6.2+.

Another note in the CR says that it's done for stacking... not sure if it was added to standalone switches. Is yours a stack or standalone?
Userlevel 3
Drew C. wrote:

Hello friend, what version was installed previously? I'm assuming you were happy with the rate in that previous version?
The LED blink rate was slowed under CR xos0059128, which is included in your patch version.
It's also in EXOS 16.1.1, EXOS 15.7.1+, and EXOS 15.6.2+.

Another note in the CR says that it's done for stacking... not sure if it was added to standalone switches. Is yours a stack or standalone?

Hi!

15.4.1.3-patch1-1 was installed previosly. I use two standalone switches whish is MLAG peers.

It's reaaly strange, but I have misterious fealing that it's not the XOS problem. Something happen in network.

to be continued...
Funny I just did this last night and I see the same crazy blinking and high cpu usage. Chunky response in the CLI and if you do a "show stacking" command forget about it you'll have to restart your telnet session. I shouldn't have touched anything I guess.
Userlevel 3
Jed wrote:

Funny I just did this last night and I see the same crazy blinking and high cpu usage. Chunky response in the CLI and if you do a "show stacking" command forget about it you'll have to restart your telnet session. I shouldn't have touched anything I guess.

My switches look great. CPU usage is low, everething is good. But LED blinking looks pretty creepy.
Userlevel 3
So, I've done a lot of test. If no traffic in port it doesn't blink, if the traffic is in port it blinks in sync with the others.
Userlevel 7
eyeV wrote:

So, I've done a lot of test. If no traffic in port it doesn't blink, if the traffic is in port it blinks in sync with the others.

Thanks for looking at this. I think it would be a good idea to open a ticket with GTAC so that we can route this to engineering for further investigation.
Userlevel 7
I've had a chat with one of the Summit developers. Here's what I learned:

The X670’s LED activity is updated by software, and not directly by the Broadcom chip like in other EXOS platforms. This software process runs at a period of one second. In each period, it checks if RX and TX counters have changed on each port. Since the counter thread runs at a “slow” period, the LED processor is very likely to see link activity on every port (if any) at every scan. So, it blinks the LEDs at the same time. In summary, this is a hardware limitation on X670 that we are unfortunately unable to do anything about.

There were some changes to the timing and duration of the LED blinking, but it is still the same idea as before. Previously, the scan ran a little faster but still blinked all LEDs at the same time on ports with activity.
Userlevel 3
Drew C. wrote:

I've had a chat with one of the Summit developers. Here's what I learned:

The X670’s LED activity is updated by software, and not directly by the Broadcom chip like in other EXOS platforms. This software process runs at a period of one second. In each period, it checks if RX and TX counters have changed on each port. Since the counter thread runs at a “slow” period, the LED processor is very likely to see link activity on every port (if any) at every scan. So, it blinks the LEDs at the same time. In summary, this is a hardware limitation on X670 that we are unfortunately unable to do anything about.

There were some changes to the timing and duration of the LED blinking, but it is still the same idea as before. Previously, the scan ran a little faster but still blinked all LEDs at the same time on ports with activity.

Thanks a lot Drew!

Hmm... I think I haven't noticed this blinking previously )

Reply