topic Re: Amber LED and Operate Routing, what does it mean? in ExtremeSwitching (VSP/Fabric Engine)

Amber LED and Operate Routing, what does it mean?

Anonymous — Thu, 11 Feb 2021 05:56:00 GMT

Hi,

Have a single port that is showing a solid amber LED, yet it seems to be working fine.

It is configured as an SMLT across a cluster pair. Pulling one or the other link on either cluster, the switch on the end carries on working. The same port on both the cluster members is showing the amber LED.

What is odd about it is if you shutdown the interface and re-enable it again the LEDs look normal i.e. green and flashing. When you do the other side the same happens, but at some point, not sure where in the sequence they go amber again,

I can’t see anything on the ports that is showing an error. Equally if I compare the config to another port that is working fine they are the same.

This seems to be what the LED colours refer to:

The only thing I have noticed is from the output below that port 3/1, the port with the issue shows ‘Operate Routing’ disable, where as all the others show it enabled - have not idea what this means?

show interfaces gigabitEthernet config 3/1-3/13
************************************************************************************
                Command Execution Time: Wed Feb 10 21:35:07 2021 GMT
************************************************************************************
====================================================================================================
                                  Port Config
====================================================================================================
PORT                         DIFF-SERV    QOS   MLT   VENDOR
NUM      TYPE                EN    TYPE   LVL   ID    NAME
----------------------------------------------------------------------------------------------------
3/1      GbicSx              true  core   1     301   AVAGO
3/2      10GbSR              true  core   1     302   AVAGO
3/3      10GbSR              true  core   1     303   FINISAR CORP.
3/4      GbicSx              true  core   1     304   AVAGO
3/5      GbicSx              true  core   1     0     AVAGO
3/6      GbicSx              true  core   1     306   AVAGO
3/7      GbicSx              true  core   1     307   AVAGO
3/8      GbicSx              true  core   1     308   AVAGO
3/9      GbicSx              true  core   1     309   AVAGO
3/10     GbicSx              true  core   1     310   AVAGO
3/11     GbicSx              true  core   1     311   AVAGO
3/12     GbicSx              true  core   1     312   AVAGO
3/13     GbicSx              true  core   1     0     AVAGO

PORT     ADMIN   OPERATE AUTO     ACCESS-SERV  RMON     FLEX-UNI ADMIN   FEC-APPLICABLE   OPERATE
NUM      ROUTING ROUTING RECOVER  EN                             FEC                      FEC
----------------------------------------------------------------------------------------------------
3/1      Enable  Disable Disable  false        Disable  Disable  Auto    Not Applicable   N/A
3/2      Enable  Enable  Disable  false        Disable  Disable  Auto    Not Applicable   N/A
3/3      Enable  Enable  Disable  false        Disable  Disable  Auto    Not Applicable   N/A
3/4      Enable  Enable  Disable  false        Disable  Disable  Auto    Not Applicable   N/A
3/5      Enable  Enable  Disable  false        Disable  Disable  Auto    Not Applicable   N/A
3/6      Enable  Enable  Disable  false        Disable  Disable  Auto    Not Applicable   N/A
3/7      Enable  Enable  Disable  false        Disable  Disable  Auto    Not Applicable   N/A
3/8      Enable  Enable  Disable  false        Disable  Disable  Auto    Not Applicable   N/A
3/9      Enable  Enable  Disable  false        Disable  Disable  Auto    Not Applicable   N/A
3/10     Enable  Enable  Disable  false        Disable  Disable  Auto    Not Applicable   N/A
3/11     Enable  Enable  Disable  false        Disable  Disable  Auto    Not Applicable   N/A
3/12     Enable  Enable  Disable  false        Disable  Disable  Auto    Not Applicable   N/A

Configuration for example between ports 3/1 and 3/4 looks the same

config terminal
interface GigabitEthernet 3/1
   encapsulation dot1q
exit
vlan members remove 1 1/1-1/18,3/1-3/24,4/1-4/24 portmember
vlan members 626 3/1 portmember
vlan members 627 3/1-3/2 portmember
vlan members 935 3/1 portmember
vlan members 1003 3/1-3/5 portmember
vlan members 2002 3/1-3/13,3/21 portmember
interface GigabitEthernet 3/1
   untagged-frames-discard
   default-vlan-id 0
   name "PorLANst119A"
   no shutdown
   slpp packet-rx
   slpp packet-rx-threshold 500
   lacp key 301 aggregation enable
   lacp enable
   no spanning-tree mstp  force-port-state enable
   sflow sampling-rate 8192
   sflow collector 1
exit
qos queue-profile 1 member add 1/1-1/18,3/1-3/24,4/1-4/24
end

interface GigabitEthernet 3/4
   encapsulation dot1q
exit
vlan members remove 1 1/1-1/18,3/1-3/24,4/1-4/24 portmember
vlan members 632 3/4 portmember
vlan members 832 3/4 portmember
vlan members 932 3/4 portmember
vlan members 1003 3/1-3/5 portmember
vlan members 1100 3/4,3/6,3/8-3/9,3/21 portmember
vlan members 2002 3/1-3/13,3/21 portmember
interface GigabitEthernet 3/4
   untagged-frames-discard
   default-vlan-id 0
   name "PorLANSt129A"
   no shutdown
   slpp packet-rx
   slpp packet-rx-threshold 500
   lacp key 304 aggregation enable
   lacp enable
   no spanning-tree mstp  force-port-state enable
   sflow sampling-rate 8192
   sflow collector 1
exit
qos queue-profile 1 member add 1/1-1/18,3/1-3/24,4/1-4/24
end

Many thanks in advance.

Re: Amber LED and Operate Routing, what does it mean?

Dan15 — Thu, 11 Feb 2021 18:25:43 GMT

Are this ports forming an SMLT towards a L2? A came across this issue with having VLACP configured on one side, but not on the other. So the links is physically up, but VLACP (not LACP) protocol prevents any packets sent on the port.
I can see from your config that you have LACP configured. Was that intentional? LACP is not needed in an SMLT setup.

Re: Amber LED and Operate Routing, what does it mean?

Anonymous — Thu, 11 Feb 2021 20:55:39 GMT

Hi Dany,

Thanks for posting back.

LACP was intentional.

This might be old practise coming up the Enterasys / Extreme route but only ever nailed (non LACP) LAGs on critical links like the ISC. When talking the servers or edge switches using LACP helped minimise the potential for issues in that interfaces are negotiated and happy before forming the LAG, plus it is typically enabled when talking to servers.

Equally though I have seen issues with that same negotiation and interfaces potentially coming up before the LAG has formed and creating a loop, but the habit has stuck nonetheless.

I have read that using MLT’s without LACP is the norm, but predominantly when talking to other Extreme devices like BOSS, VOSS and EXOS switches.

In this case I am use EOS switches, which wasn’t on that list, so treated them as an external vendor and went for the LACP approach. Also, I believe there where some problems when not using LACP to these switches in the past?

SLPP has been introduced so that will obviously help any miss-configuration but without being able to use SLPP guard in EOS, you can isolate the switch if a loop is introduced on the front end switch. The EOS switch should be correctly configured with STP to prevent this, but you don’t always have that luxury to mandate its been implemented correctly.

Anyway, that is part of the logic behind using LACP. Have seen it used with the same exact same architecture in other projects and seems to work fine, but interested to hear what others do. If none LACP MLT’s is always used as the norm, then I can trust that approach.

It is a L2 configuration (just an edge switch) the SMLT is connecting to. There are about a dozen switches at this time connected in the same way, all without issue.

It’s just this port that is displaying an oddity. I can’t see anything in the switch outputs that show it different from any other port, other then the ‘Operate Routing’ disabled shown above.

Both the switches in the cluster are configured the same, no VLACP has been configured but will look into that to be sure.

Does seem something like that, some underlying protocol is actioning against the port, but can’t figure out a command for to tell me what?

Thanks

Re: Amber LED and Operate Routing, what does it mean?

Dan15 — Fri, 12 Feb 2021 00:41:18 GMT

I do not know about third party devices with SMLT, but you are probably right that LACP needs to be used.

If an underlying protocol is interfering, it should be in the log. Have you checked them?

show log file tail

You can also try the alarm database:

show alarm database

There is an article about SMLT with LACP:

https://extremeportal.force.com/ExtrArticleDetail?an=000083501&q=smlt

You maybe have a misconfiguration on VLAN level creating a loop. Are the I-SIDs correctly on both cluster members for the Edge VLANs?

Re: Amber LED and Operate Routing, what does it mean?

Anonymous — Fri, 12 Feb 2021 04:01:11 GMT

All looks ok from what I can tell.

No alarms in the database, no messages in the log or in detailed log, config is identical on both sides:

DC1-Core2:1#% cfg || 3/1
           alias% show running-config -bi || 3/1
config terminal
interface GigabitEthernet 3/1
   encapsulation dot1q
exit
vlan members remove 1 1/1-1/18,3/1-3/24,4/1-4/24 portmember
vlan members 626 3/1 portmember
vlan members 627 3/1-3/2 portmember
vlan members 935 3/1 portmember
vlan members 1003 3/1-3/5 portmember
vlan members 2002 3/1-3/13,3/21 portmember
interface GigabitEthernet 3/1
   untagged-frames-discard
   default-vlan-id 0
   name "PorLANst119A"
   no shutdown
   slpp packet-rx
   slpp packet-rx-threshold 50
   lacp key 301 aggregation enable
   lacp enable
   no spanning-tree mstp  force-port-state enable
   sflow sampling-rate 8192
   sflow collector 1
exit
qos queue-profile 1 member add 1/1-1/18,3/1-3/24,4/1-4/24
end

DC1-Core2:1#% cfg || mlt 301
           alias% show running-config -bi || mlt 301
config terminal
mlt 301 enable name "PorLANst119A"
vlan mlt 626 301
vlan mlt 627 301
vlan mlt 935 301
vlan mlt 1003 301
vlan mlt 2002 301
interface mlt 301
   smlt
   lacp enable key 301
exit
end

DC2-Core2:1#% cfg || 3/1
           alias% show running-config -bi || 3/1
config terminal
interface GigabitEthernet 3/1
   encapsulation dot1q
exit
vlan members remove 1 1/1-1/18,3/1-3/24,4/1-4/24 portmember
vlan members 626 3/1 portmember
vlan members 627 3/1-3/2 portmember
vlan members 935 3/1 portmember
vlan members 1003 3/1-3/5 portmember
vlan members 2002 3/1-3/13,3/21 portmember
interface GigabitEthernet 3/1
   untagged-frames-discard
   default-vlan-id 0
   name "PorLANst119A"
   no shutdown
   slpp packet-rx
   slpp packet-rx-threshold 500
   lacp key 301 aggregation enable
   lacp enable
   no spanning-tree mstp  force-port-state enable
   sflow sampling-rate 8192
   sflow collector 1
exit
qos queue-profile 1 member add 1/1-1/18,3/1-3/24,4/1-4/24
end

DC2-Core2:1#% cfg || mlt 301
           alias% show running-config -bi || mlt 301
config terminal
mlt 301 enable name "PorLANst119A"
vlan mlt 626 301
vlan mlt 627 301
vlan mlt 935 301
vlan mlt 1003 301
vlan mlt 2002 301
interface mlt 301
   smlt
   lacp enable key 301
exit
end

The only thing that I can find in any log files, show outputs, configs that is different between working and none working is still that ‘Operate Routing’, showing disabled where all the others show enabled as per below?

DC2-Core2:1#% show interfaces gigabitEthernet config 3/1-3/4
************************************************************************************
                Command Execution Time: Thu Feb 11 19:56:31 2021 GMT
************************************************************************************
====================================================================================================
                                  Port Config
====================================================================================================
PORT                         DIFF-SERV    QOS   MLT   VENDOR
NUM      TYPE                EN    TYPE   LVL   ID    NAME
----------------------------------------------------------------------------------------------------
3/1      GbicSx              true  core   1     301   AVAGO
3/2      10GbSR              true  core   1     302   AVAGO
3/3      10GbSR              true  core   1     303   FINISAR CORP.
3/4      GbicSx              true  core   1     304   AVAGO

PORT     ADMIN   OPERATE AUTO     ACCESS-SERV  RMON     FLEX-UNI ADMIN   FEC-APPLICABLE   OPERATE
NUM      ROUTING ROUTING RECOVER  EN                             FEC                      FEC
----------------------------------------------------------------------------------------------------
3/1      Enable  Disable Disable  false        Disable  Disable  Auto    Not Applicable   N/A
3/2      Enable  Enable  Disable  false        Disable  Disable  Auto    Not Applicable   N/A
3/3      Enable  Enable  Disable  false        Disable  Disable  Auto    Not Applicable   N/A
3/4      Enable  Enable  Disable  false        Disable  Disable  Auto    Not Applicable   N/A

On an additional question about LED’s, do you know if there is a command to stop them all flashing?

Have a couple of 8404’s out the box where two all ports are flashing, other two are not and have no idea why?

Many thanks

Re: Amber LED and Operate Routing, what does it mean?

Dan15 — Fri, 12 Feb 2021 15:44:28 GMT

I have checked the “Operrouting” on my side. All UNI ports with a VLAN and an IP address attached to this VLAN are on Operrouting enabled. The NNI ports with only the BB VLANs 4051-4052 are on Operrouting disabled. This makes sense. The doc says: Shows the routing status of the port.

The only thing I know about the flashing port LEDs on VOSS is when they are administratively shut. By default they are all shut.

Do you have configured the SMLT VLANs with the same I-SID on both cluster members so they are connected via the virtual IST? (show mlt i-sid)

Maybe there is a problem on L1 with the SFP? Do you have something in the DDI informations? (show pluggable-optical-modules detail)

Re: Amber LED and Operate Routing, what does it mean?

Anonymous — Fri, 12 Feb 2021 17:13:34 GMT

Hi Dany,

Thanks for putting the time in to look at it, much appreciated.

Here is the output of the port with the issue and one that is OK, so look fairly the same:


====================================================================================================
                              Pluggable Optical Module Info 3/1 Detail
====================================================================================================
Port: 3/1
Type: GbicSx
DDM Supported : TRUE
Vendor Name  : AVAGO                Partnumber : AFBR-5715PZ       SKU:
Vendor REV   :                      Vendor SN  : AM1152SXE9D
Vendor Date  : 12/29/11
Wavelength   : 850.00 nm

Digital Diagnostic Interface Supported

Optics Status              : Ok
Calibration             : Internal
RX Power Measurement    : Average
Auxiliary 1 Monitoring  : Not Implemented
Auxiliary 2 Monitoring  : Not Implemented

----------------------------------------------------------------------------------------------------
              LOW_ALARM LOW_WARN    ACTUAL  HIGH_WARN HIGH_ALARM THRESHOLD
              THRESHOLD THRESHOLD   VALUE   THRESHOLD THRESHOLD  STATUS
----------------------------------------------------------------------------------------------------
Temp(C)        -40.0     -10.0      32.1875   85.0     100.0     Normal
Voltage(V)       2.9700    3.1349    3.2877    3.4650    3.6300  Normal
Bias(mA)         2.0       2.0       6.1580   10.0      12.0     Normal
TxPower(dBm)   -12.3000  -10.8000   -5.5000    0.0       0.4000  Normal
RxPower(dBm)   -40.0     -17.0      -6.1000    0.0       2.5000  Normal

Extreme Extended Diagnostic Information
----------------------------------------------------------------------------------------------------
Not Available


====================================================================================================
                              Pluggable Optical Module Info 3/4 Detail
====================================================================================================
Port: 3/4
Type: GbicSx
DDM Supported : TRUE
Vendor Name  : AVAGO                Partnumber : AFBR-5715PZ       SKU:
Vendor REV   :                      Vendor SN  : AM1152SXBY9
Vendor Date  : 12/28/11
Wavelength   : 850.00 nm

Digital Diagnostic Interface Supported

Optics Status              : Ok
Calibration             : Internal
RX Power Measurement    : Average
Auxiliary 1 Monitoring  : Not Implemented
Auxiliary 2 Monitoring  : Not Implemented

----------------------------------------------------------------------------------------------------
              LOW_ALARM LOW_WARN    ACTUAL  HIGH_WARN HIGH_ALARM THRESHOLD
              THRESHOLD THRESHOLD   VALUE   THRESHOLD THRESHOLD  STATUS
----------------------------------------------------------------------------------------------------
Temp(C)        -40.0     -10.0      34.7773   85.0     100.0     Normal
Voltage(V)       2.9700    3.1349    3.2813    3.4650    3.6300  Normal
Bias(mA)         2.0       2.0       6.4640   10.0      12.0     Normal
TxPower(dBm)   -12.3000  -10.8000   -5.4000    0.0       0.4000  Normal
RxPower(dBm)   -40.0     -17.0      -6.7000    0.0       2.5000  Normal

Extreme Extended Diagnostic Information
----------------------------------------------------------------------------------------------------
Not Available
====================================================================================================

Here is the show mlt i-sid. I used an MLT that match the port i.e. 301 is port 3/1, 304 is 3/4, 423 is 4/23 and so on

DC1-Core2:1#% show mlt i-sid
************************************************************************************
                Command Execution Time: Fri Feb 12 09:03:19 2021 GMT
************************************************************************************
==============================================================================================================
                                                MLT Isid Info
==============================================================================================================
              ISID                   ISID                        ISID
MLTID IFINDEX ID       VLANID C-VID  TYPE     ORIGIN             NAME                             BPDU
--------------------------------------------------------------------------------------------------------------
301   6444    12110626 626    N/A    CVLAN    CONFIG             ISID-12110626
301   6444    12110627 627    N/A    CVLAN    CONFIG             ISID-12110627
301   6444    12110935 935    N/A    CVLAN    CONFIG             ISID-12110935
301   6444    12111003 1003   N/A    CVLAN    CONFIG             ISID-12111003
301   6444    12112002 2002   N/A    CVLAN    CONFIG             ISID-12112002
302   6445    12110627 627    N/A    CVLAN    CONFIG             ISID-12110627
302   6445    12110827 827    N/A    CVLAN    CONFIG             ISID-12110827
302   6445    12110927 927    N/A    CVLAN    CONFIG             ISID-12110927
302   6445    12111003 1003   N/A    CVLAN    CONFIG             ISID-12111003
302   6445    12112002 2002   N/A    CVLAN    CONFIG             ISID-12112002
303   6446    12110631 631    N/A    CVLAN    CONFIG             ISID-12110631
303   6446    12110828 828    N/A    CVLAN    CONFIG             ISID-12110828
303   6446    12110831 831    N/A    CVLAN    CONFIG             ISID-12110831
303   6446    12110931 931    N/A    CVLAN    CONFIG             ISID-12110931
303   6446    12111003 1003   N/A    CVLAN    CONFIG             ISID-12111003
303   6446    12112002 2002   N/A    CVLAN    CONFIG             ISID-12112002
304   6447    12110632 632    N/A    CVLAN    CONFIG             ISID-12110632
304   6447    12110832 832    N/A    CVLAN    CONFIG             ISID-12110832
304   6447    12110932 932    N/A    CVLAN    CONFIG             ISID-12110932
304   6447    12111003 1003   N/A    CVLAN    CONFIG             ISID-12111003
304   6447    12111100 1100   N/A    CVLAN    CONFIG             ISID-12111100
304   6447    12112002 2002   N/A    CVLAN    CONFIG             ISID-12112002

Each of the VLANs used have IP address associated, except for VLAN 2002, but this is being used on other ports without an issue:

DC1-Core2:1#% show ip interface || 626,627,935,2002
************************************************************************************
                Command Execution Time: Fri Feb 12 09:06:37 2021 GMT
************************************************************************************
====================================================================================================
                          IP Interface - GlobalRouter
====================================================================================================
INTERFACE    IP             NET            BCASTADDR  REASM    VLAN  BROUTER    IPSEC   IP
             ADDRESS        MASK           FORMAT     MAXSIZE  ID    PORT       STATE   NAME
----------------------------------------------------------------------------------------------------
Vlan626      192.168.161.146 255.255.255.240 ones       1500     626   false      disable
Vlan627      192.168.161.162 255.255.255.240 ones       1500     627   false      disable
Vlan935      192.168.102.34 255.255.255.240 ones       1500     935   false      disable

Strange right, might open a GTAC case as doesn’t seem straight forward. To me it seems like a L1 issue, primarily as all the other ports are OK and config wise I can’t see any difference.

If you have anymore ideas let me know, if I find out anything in the meantime I’ll post back.

Many thanks

Re: Amber LED and Operate Routing, what does it mean?

Dan15 — Fri, 12 Feb 2021 19:58:51 GMT

You can try to move the SFP to another slot and or replace the SFP.

I found another article and it looks very similar to your case:

https://extremeportal.force.com/ExtrArticleDetail?an=000060397&q=amber

Re: Amber LED and Operate Routing, what does it mean?

Anonymous — Wed, 17 Feb 2021 23:28:06 GMT

Issue turned out to be with the AVAGO Gbic.

So although the same where used in other slots, had been swapped with another Gbic that seemed to working in another slot and equally had been moved working from an S8 to VOSS, something about it just wasn’t compatible.

These GBIC’s turned out not to be on the Extreme approved list, and thereby replacing with Extreme badged it worked fine.

Raised with GTAC whom identified the issue and provided this link to identify supported GBICs

https://optics.extremenetworks.com/