Amber LED and Operate Routing, what does it mean?

[Anonymous]

Hi,

Have a single port that is showing a solid amber LED, yet it seems to be working fine.

It is configured as an SMLT across a cluster pair. Pulling one or the other link on either cluster, the switch on the end carries on working. The same port on both the cluster members is showing the amber LED.

What is odd about it is if you shutdown the interface and re-enable it again the LEDs look normal i.e. green and flashing. When you do the other side the same happens, but at some point, not sure where in the sequence they go amber again,

I can’t see anything on the ports that is showing an error. Equally if I compare the config to another port that is working fine they are the same.

This seems to be what the LED colours refer to:

The only thing I have noticed is from the output below that port 3/1, the port with the issue shows ‘Operate Routing’ disable, where as all the others show it enabled - have not idea what this means?

show interfaces gigabitEthernet config 3/1-3/13
************************************************************************************
                Command Execution Time: Wed Feb 10 21:35:07 2021 GMT
************************************************************************************
====================================================================================================
                                  Port Config
====================================================================================================
PORT                         DIFF-SERV    QOS   MLT   VENDOR
NUM      TYPE                EN    TYPE   LVL   ID    NAME
----------------------------------------------------------------------------------------------------
3/1      GbicSx              true  core   1     301   AVAGO
3/2      10GbSR              true  core   1     302   AVAGO
3/3      10GbSR              true  core   1     303   FINISAR CORP.
3/4      GbicSx              true  core   1     304   AVAGO
3/5      GbicSx              true  core   1     0     AVAGO
3/6      GbicSx              true  core   1     306   AVAGO
3/7      GbicSx              true  core   1     307   AVAGO
3/8      GbicSx              true  core   1     308   AVAGO
3/9      GbicSx              true  core   1     309   AVAGO
3/10     GbicSx              true  core   1     310   AVAGO
3/11     GbicSx              true  core   1     311   AVAGO
3/12     GbicSx              true  core   1     312   AVAGO
3/13     GbicSx              true  core   1     0     AVAGO

PORT     ADMIN   OPERATE AUTO     ACCESS-SERV  RMON     FLEX-UNI ADMIN   FEC-APPLICABLE   OPERATE
NUM      ROUTING ROUTING RECOVER  EN                             FEC                      FEC
----------------------------------------------------------------------------------------------------
3/1      Enable  Disable Disable  false        Disable  Disable  Auto    Not Applicable   N/A
3/2      Enable  Enable  Disable  false        Disable  Disable  Auto    Not Applicable   N/A
3/3      Enable  Enable  Disable  false        Disable  Disable  Auto    Not Applicable   N/A
3/4      Enable  Enable  Disable  false        Disable  Disable  Auto    Not Applicable   N/A
3/5      Enable  Enable  Disable  false        Disable  Disable  Auto    Not Applicable   N/A
3/6      Enable  Enable  Disable  false        Disable  Disable  Auto    Not Applicable   N/A
3/7      Enable  Enable  Disable  false        Disable  Disable  Auto    Not Applicable   N/A
3/8      Enable  Enable  Disable  false        Disable  Disable  Auto    Not Applicable   N/A
3/9      Enable  Enable  Disable  false        Disable  Disable  Auto    Not Applicable   N/A
3/10     Enable  Enable  Disable  false        Disable  Disable  Auto    Not Applicable   N/A
3/11     Enable  Enable  Disable  false        Disable  Disable  Auto    Not Applicable   N/A
3/12     Enable  Enable  Disable  false        Disable  Disable  Auto    Not Applicable   N/A

Configuration for example between ports 3/1 and 3/4  looks the same

config terminal
interface GigabitEthernet 3/1
   encapsulation dot1q
exit
vlan members remove 1 1/1-1/18,3/1-3/24,4/1-4/24 portmember
vlan members 626 3/1 portmember
vlan members 627 3/1-3/2 portmember
vlan members 935 3/1 portmember
vlan members 1003 3/1-3/5 portmember
vlan members 2002 3/1-3/13,3/21 portmember
interface GigabitEthernet 3/1
   untagged-frames-discard
   default-vlan-id 0
   name "PorLANst119A"
   no shutdown
   slpp packet-rx
   slpp packet-rx-threshold 500
   lacp key 301 aggregation enable
   lacp enable
   no spanning-tree mstp  force-port-state enable
   sflow sampling-rate 8192
   sflow collector 1
exit
qos queue-profile 1 member add 1/1-1/18,3/1-3/24,4/1-4/24
end

interface GigabitEthernet 3/4
   encapsulation dot1q
exit
vlan members remove 1 1/1-1/18,3/1-3/24,4/1-4/24 portmember
vlan members 632 3/4 portmember
vlan members 832 3/4 portmember
vlan members 932 3/4 portmember
vlan members 1003 3/1-3/5 portmember
vlan members 1100 3/4,3/6,3/8-3/9,3/21 portmember
vlan members 2002 3/1-3/13,3/21 portmember
interface GigabitEthernet 3/4
   untagged-frames-discard
   default-vlan-id 0
   name "PorLANSt129A"
   no shutdown
   slpp packet-rx
   slpp packet-rx-threshold 500
   lacp key 304 aggregation enable
   lacp enable
   no spanning-tree mstp  force-port-state enable
   sflow sampling-rate 8192
   sflow collector 1
exit
qos queue-profile 1 member add 1/1-1/18,3/1-3/24,4/1-4/24
end

Many thanks in advance.

[Anonymous]

All looks ok from what I can tell.

No alarms in the database, no messages in the log or in detailed log, config is identical on both sides:

DC1-Core2:1#% cfg || 3/1
           alias% show running-config -bi || 3/1
config terminal
interface GigabitEthernet 3/1
   encapsulation dot1q
exit
vlan members remove 1 1/1-1/18,3/1-3/24,4/1-4/24 portmember
vlan members 626 3/1 portmember
vlan members 627 3/1-3/2 portmember
vlan members 935 3/1 portmember
vlan members 1003 3/1-3/5 portmember
vlan members 2002 3/1-3/13,3/21 portmember
interface GigabitEthernet 3/1
   untagged-frames-discard
   default-vlan-id 0
   name "PorLANst119A"
   no shutdown
   slpp packet-rx
   slpp packet-rx-threshold 50
   lacp key 301 aggregation enable
   lacp enable
   no spanning-tree mstp  force-port-state enable
   sflow sampling-rate 8192
   sflow collector 1
exit
qos queue-profile 1 member add 1/1-1/18,3/1-3/24,4/1-4/24
end

DC1-Core2:1#% cfg || mlt 301
           alias% show running-config -bi || mlt 301
config terminal
mlt 301 enable name "PorLANst119A"
vlan mlt 626 301
vlan mlt 627 301
vlan mlt 935 301
vlan mlt 1003 301
vlan mlt 2002 301
interface mlt 301
   smlt
   lacp enable key 301
exit
end

DC2-Core2:1#% cfg || 3/1
           alias% show running-config -bi || 3/1
config terminal
interface GigabitEthernet 3/1
   encapsulation dot1q
exit
vlan members remove 1 1/1-1/18,3/1-3/24,4/1-4/24 portmember
vlan members 626 3/1 portmember
vlan members 627 3/1-3/2 portmember
vlan members 935 3/1 portmember
vlan members 1003 3/1-3/5 portmember
vlan members 2002 3/1-3/13,3/21 portmember
interface GigabitEthernet 3/1
   untagged-frames-discard
   default-vlan-id 0
   name "PorLANst119A"
   no shutdown
   slpp packet-rx
   slpp packet-rx-threshold 500
   lacp key 301 aggregation enable
   lacp enable
   no spanning-tree mstp  force-port-state enable
   sflow sampling-rate 8192
   sflow collector 1
exit
qos queue-profile 1 member add 1/1-1/18,3/1-3/24,4/1-4/24
end

DC2-Core2:1#% cfg || mlt 301
           alias% show running-config -bi || mlt 301
config terminal
mlt 301 enable name "PorLANst119A"
vlan mlt 626 301
vlan mlt 627 301
vlan mlt 935 301
vlan mlt 1003 301
vlan mlt 2002 301
interface mlt 301
   smlt
   lacp enable key 301
exit
end

The only thing that I can find in any log files, show outputs, configs that is different between working and none working is still that ‘Operate Routing’, showing disabled where all the others show enabled as per below?

DC2-Core2:1#% show interfaces gigabitEthernet config 3/1-3/4
************************************************************************************
                Command Execution Time: Thu Feb 11 19:56:31 2021 GMT
************************************************************************************
====================================================================================================
                                  Port Config
====================================================================================================
PORT                         DIFF-SERV    QOS   MLT   VENDOR
NUM      TYPE                EN    TYPE   LVL   ID    NAME
----------------------------------------------------------------------------------------------------
3/1      GbicSx              true  core   1     301   AVAGO
3/2      10GbSR              true  core   1     302   AVAGO
3/3      10GbSR              true  core   1     303   FINISAR CORP.
3/4      GbicSx              true  core   1     304   AVAGO

PORT     ADMIN   OPERATE AUTO     ACCESS-SERV  RMON     FLEX-UNI ADMIN   FEC-APPLICABLE   OPERATE
NUM      ROUTING ROUTING RECOVER  EN                             FEC                      FEC
----------------------------------------------------------------------------------------------------
3/1      Enable  Disable Disable  false        Disable  Disable  Auto    Not Applicable   N/A
3/2      Enable  Enable  Disable  false        Disable  Disable  Auto    Not Applicable   N/A
3/3      Enable  Enable  Disable  false        Disable  Disable  Auto    Not Applicable   N/A
3/4      Enable  Enable  Disable  false        Disable  Disable  Auto    Not Applicable   N/A

On an additional question about LED’s, do you know if there is a command to stop them all flashing?

Have a couple of 8404’s out the box where two all ports are flashing, other two are not and have no idea why?

Many thanks

[Dan15]

I do not know about third party devices with SMLT, but you are probably right that LACP needs to be used.

If an underlying protocol is interfering, it should be in the log. Have you checked them?

show log file tail

You can also try the alarm database:

show alarm database

There is an article about SMLT with LACP:

https://extremeportal.force.com/ExtrArticleDetail?an=000083501&q=smlt

You maybe have a misconfiguration on VLAN level creating a loop. Are the I-SIDs correctly on both cluster members for the Edge VLANs?

[Anonymous]

Hi Dany,

Thanks for posting back.

LACP was intentional.

This might be old practise coming up the Enterasys / Extreme route but only ever nailed (non LACP) LAGs on critical links like the ISC. When talking the servers or edge switches using LACP helped minimise the potential for issues in that interfaces are negotiated and happy before forming the LAG, plus it is typically enabled when talking to servers.

Equally though I have seen issues with that same negotiation and interfaces potentially coming up before the LAG has formed and creating a loop, but the habit has stuck nonetheless.

I have read that using MLT’s without LACP is the norm, but predominantly when talking to other Extreme devices like BOSS, VOSS and EXOS switches.

In this case I am use EOS switches, which wasn’t on that list, so treated them as an external vendor and went for the LACP approach. Also, I believe there where some problems when not using LACP to these switches in the past?

SLPP has been introduced so that will obviously help any miss-configuration but without being able to use SLPP guard in EOS, you can isolate the switch if a loop is introduced on the front end switch. The EOS switch should be correctly configured with STP to prevent this, but you don’t always have that luxury to mandate its been implemented correctly.

Anyway, that is part of the logic behind using LACP. Have seen it used with the same exact same architecture in other projects and seems to work fine, but interested to hear what others do. If none LACP MLT’s is always used as the norm, then I can trust that approach.

It is a L2 configuration (just an edge switch) the SMLT is connecting to. There are about a dozen switches at this time connected in the same way, all without issue.

It’s just this port that is displaying an oddity. I can’t see anything in the switch outputs that show it different from any other port, other then the ‘Operate Routing’ disabled shown above.

Both the switches in the cluster are configured the same, no VLACP has been configured but will look into that to be sure.

Does seem something like that, some underlying protocol is actioning against the port, but can’t figure out a command for to tell me what?

Thanks

[Dan15]

Are this ports forming an SMLT towards a L2? A came across this issue with having VLACP configured on one side, but not on the other. So the links is physically up, but VLACP (not LACP) protocol prevents any packets sent on the port.
I can see from your config that you have LACP configured. Was that intentional? LACP is not needed in an SMLT setup.