Solved

Amber LED and Operate Routing, what does it mean?

  • 10 February 2021
  • 8 replies
  • 77 views

Userlevel 6
Badge

Hi,

Have a single port that is showing a solid amber LED, yet it seems to be working fine.

It is configured as an SMLT across a cluster pair. Pulling one or the other link on either cluster, the switch on the end carries on working. The same port on both the cluster members is showing the amber LED.

What is odd about it is if you shutdown the interface and re-enable it again the LEDs look normal i.e. green and flashing. When you do the other side the same happens, but at some point, not sure where in the sequence they go amber again,

I can’t see anything on the ports that is showing an error. Equally if I compare the config to another port that is working fine they are the same.

This seems to be what the LED colours refer to:

 

The only thing I have noticed is from the output below that port 3/1, the port with the issue shows ‘Operate Routing’ disable, where as all the others show it enabled - have not idea what this means?

show interfaces gigabitEthernet config 3/1-3/13
************************************************************************************
Command Execution Time: Wed Feb 10 21:35:07 2021 GMT
************************************************************************************
====================================================================================================
Port Config
====================================================================================================
PORT DIFF-SERV QOS MLT VENDOR
NUM TYPE EN TYPE LVL ID NAME
----------------------------------------------------------------------------------------------------
3/1 GbicSx true core 1 301 AVAGO
3/2 10GbSR true core 1 302 AVAGO
3/3 10GbSR true core 1 303 FINISAR CORP.
3/4 GbicSx true core 1 304 AVAGO
3/5 GbicSx true core 1 0 AVAGO
3/6 GbicSx true core 1 306 AVAGO
3/7 GbicSx true core 1 307 AVAGO
3/8 GbicSx true core 1 308 AVAGO
3/9 GbicSx true core 1 309 AVAGO
3/10 GbicSx true core 1 310 AVAGO
3/11 GbicSx true core 1 311 AVAGO
3/12 GbicSx true core 1 312 AVAGO
3/13 GbicSx true core 1 0 AVAGO

PORT ADMIN OPERATE AUTO ACCESS-SERV RMON FLEX-UNI ADMIN FEC-APPLICABLE OPERATE
NUM ROUTING ROUTING RECOVER EN FEC FEC
----------------------------------------------------------------------------------------------------
3/1 Enable Disable Disable false Disable Disable Auto Not Applicable N/A
3/2 Enable Enable Disable false Disable Disable Auto Not Applicable N/A
3/3 Enable Enable Disable false Disable Disable Auto Not Applicable N/A
3/4 Enable Enable Disable false Disable Disable Auto Not Applicable N/A
3/5 Enable Enable Disable false Disable Disable Auto Not Applicable N/A
3/6 Enable Enable Disable false Disable Disable Auto Not Applicable N/A
3/7 Enable Enable Disable false Disable Disable Auto Not Applicable N/A
3/8 Enable Enable Disable false Disable Disable Auto Not Applicable N/A
3/9 Enable Enable Disable false Disable Disable Auto Not Applicable N/A
3/10 Enable Enable Disable false Disable Disable Auto Not Applicable N/A
3/11 Enable Enable Disable false Disable Disable Auto Not Applicable N/A
3/12 Enable Enable Disable false Disable Disable Auto Not Applicable N/A

Configuration for example between ports 3/1 and 3/4  looks the same

config terminal
interface GigabitEthernet 3/1
encapsulation dot1q
exit
vlan members remove 1 1/1-1/18,3/1-3/24,4/1-4/24 portmember
vlan members 626 3/1 portmember
vlan members 627 3/1-3/2 portmember
vlan members 935 3/1 portmember
vlan members 1003 3/1-3/5 portmember
vlan members 2002 3/1-3/13,3/21 portmember
interface GigabitEthernet 3/1
untagged-frames-discard
default-vlan-id 0
name "PorLANst119A"
no shutdown
slpp packet-rx
slpp packet-rx-threshold 500
lacp key 301 aggregation enable
lacp enable
no spanning-tree mstp force-port-state enable
sflow sampling-rate 8192
sflow collector 1
exit
qos queue-profile 1 member add 1/1-1/18,3/1-3/24,4/1-4/24
end
interface GigabitEthernet 3/4
encapsulation dot1q
exit
vlan members remove 1 1/1-1/18,3/1-3/24,4/1-4/24 portmember
vlan members 632 3/4 portmember
vlan members 832 3/4 portmember
vlan members 932 3/4 portmember
vlan members 1003 3/1-3/5 portmember
vlan members 1100 3/4,3/6,3/8-3/9,3/21 portmember
vlan members 2002 3/1-3/13,3/21 portmember
interface GigabitEthernet 3/4
untagged-frames-discard
default-vlan-id 0
name "PorLANSt129A"
no shutdown
slpp packet-rx
slpp packet-rx-threshold 500
lacp key 304 aggregation enable
lacp enable
no spanning-tree mstp force-port-state enable
sflow sampling-rate 8192
sflow collector 1
exit
qos queue-profile 1 member add 1/1-1/18,3/1-3/24,4/1-4/24
end

Many thanks in advance.

 

icon

Best answer by Martin Flammia 17 February 2021, 16:28

Issue turned out to be with the AVAGO Gbic.

So although the same where used in other slots, had been swapped with another Gbic that seemed to working in another slot and equally had been moved working from an S8 to VOSS, something about it just wasn’t compatible.

These GBIC’s turned out not to be on the Extreme approved list, and thereby replacing with Extreme badged it worked fine.

Raised with GTAC whom identified the issue and provided this link to identify supported GBICs

https://optics.extremenetworks.com/

View original

This topic has been closed for comments

8 replies

Userlevel 2

Are this ports forming an SMLT towards a L2? A came across this issue with having VLACP configured on one side, but not on the other. So the links is physically up, but VLACP (not LACP) protocol prevents any packets sent on the port.
I can see from your config that you have LACP configured. Was that intentional? LACP is not needed in an SMLT setup. 

Userlevel 6
Badge

Hi Dany,

Thanks for posting back.

LACP was intentional.

This might be old practise coming up the Enterasys / Extreme route but only ever nailed (non LACP) LAGs on critical links like the ISC. When talking the servers or edge switches using LACP helped minimise the potential for issues in that interfaces are negotiated and happy before forming the LAG, plus it is typically enabled when talking to servers.

Equally though I have seen issues with that same negotiation and interfaces potentially coming up before the LAG has formed and creating a loop, but the habit has stuck nonetheless.

I have read that using MLT’s without LACP is the norm, but predominantly when talking to other Extreme devices like BOSS, VOSS and EXOS switches.

In this case I am use EOS switches, which wasn’t on that list, so treated them as an external vendor and went for the LACP approach. Also, I believe there where some problems when not using LACP to these switches in the past?

SLPP has been introduced so that will obviously help any miss-configuration but without being able to use SLPP guard in EOS, you can isolate the switch if a loop is introduced on the front end switch. The EOS switch should be correctly configured with STP to prevent this, but you don’t always have that luxury to mandate its been implemented correctly.

Anyway, that is part of the logic behind using LACP. Have seen it used with the same exact same architecture in other projects and seems to work fine, but interested to hear what others do. If none LACP MLT’s is always used as the norm, then I can trust that approach.

It is a L2 configuration (just an edge switch) the SMLT is connecting to. There are about a dozen switches at this time connected in the same way, all without issue.

It’s just this port that is displaying an oddity. I can’t see anything in the switch outputs that show it different from any other port, other then the ‘Operate Routing’ disabled shown above.

Both the switches in the cluster are configured the same, no VLACP has been configured but will look into that to be sure.

Does seem something like that, some underlying protocol is actioning against the port, but can’t figure out a command for to tell me what?

Thanks

Userlevel 2

I do not know about third party devices with SMLT, but you are probably right that LACP needs to be used.

If an underlying protocol is interfering, it should be in the log. Have you checked them?

show log file tail

You can also try the alarm database:

show alarm database

There is an article about SMLT with LACP:

https://extremeportal.force.com/ExtrArticleDetail?an=000083501&q=smlt

You maybe have a misconfiguration on VLAN level creating a loop. Are the I-SIDs correctly on both cluster members for the Edge VLANs?

 

Userlevel 6
Badge

All looks ok from what I can tell.

No alarms in the database, no messages in the log or in detailed log, config is identical on both sides:

DC1-Core2:1#% cfg || 3/1
alias% show running-config -bi || 3/1
config terminal
interface GigabitEthernet 3/1
encapsulation dot1q
exit
vlan members remove 1 1/1-1/18,3/1-3/24,4/1-4/24 portmember
vlan members 626 3/1 portmember
vlan members 627 3/1-3/2 portmember
vlan members 935 3/1 portmember
vlan members 1003 3/1-3/5 portmember
vlan members 2002 3/1-3/13,3/21 portmember
interface GigabitEthernet 3/1
untagged-frames-discard
default-vlan-id 0
name "PorLANst119A"
no shutdown
slpp packet-rx
slpp packet-rx-threshold 50
lacp key 301 aggregation enable
lacp enable
no spanning-tree mstp force-port-state enable
sflow sampling-rate 8192
sflow collector 1
exit
qos queue-profile 1 member add 1/1-1/18,3/1-3/24,4/1-4/24
end

DC1-Core2:1#% cfg || mlt 301
alias% show running-config -bi || mlt 301
config terminal
mlt 301 enable name "PorLANst119A"
vlan mlt 626 301
vlan mlt 627 301
vlan mlt 935 301
vlan mlt 1003 301
vlan mlt 2002 301
interface mlt 301
smlt
lacp enable key 301
exit
end
DC2-Core2:1#% cfg || 3/1
alias% show running-config -bi || 3/1
config terminal
interface GigabitEthernet 3/1
encapsulation dot1q
exit
vlan members remove 1 1/1-1/18,3/1-3/24,4/1-4/24 portmember
vlan members 626 3/1 portmember
vlan members 627 3/1-3/2 portmember
vlan members 935 3/1 portmember
vlan members 1003 3/1-3/5 portmember
vlan members 2002 3/1-3/13,3/21 portmember
interface GigabitEthernet 3/1
untagged-frames-discard
default-vlan-id 0
name "PorLANst119A"
no shutdown
slpp packet-rx
slpp packet-rx-threshold 500
lacp key 301 aggregation enable
lacp enable
no spanning-tree mstp force-port-state enable
sflow sampling-rate 8192
sflow collector 1
exit
qos queue-profile 1 member add 1/1-1/18,3/1-3/24,4/1-4/24
end

DC2-Core2:1#% cfg || mlt 301
alias% show running-config -bi || mlt 301
config terminal
mlt 301 enable name "PorLANst119A"
vlan mlt 626 301
vlan mlt 627 301
vlan mlt 935 301
vlan mlt 1003 301
vlan mlt 2002 301
interface mlt 301
smlt
lacp enable key 301
exit
end

The only thing that I can find in any log files, show outputs, configs that is different between working and none working is still that ‘Operate Routing’, showing disabled where all the others show enabled as per below?

DC2-Core2:1#% show interfaces gigabitEthernet config 3/1-3/4
************************************************************************************
Command Execution Time: Thu Feb 11 19:56:31 2021 GMT
************************************************************************************
====================================================================================================
Port Config
====================================================================================================
PORT DIFF-SERV QOS MLT VENDOR
NUM TYPE EN TYPE LVL ID NAME
----------------------------------------------------------------------------------------------------
3/1 GbicSx true core 1 301 AVAGO
3/2 10GbSR true core 1 302 AVAGO
3/3 10GbSR true core 1 303 FINISAR CORP.
3/4 GbicSx true core 1 304 AVAGO

PORT ADMIN OPERATE AUTO ACCESS-SERV RMON FLEX-UNI ADMIN FEC-APPLICABLE OPERATE
NUM ROUTING ROUTING RECOVER EN FEC FEC
----------------------------------------------------------------------------------------------------
3/1 Enable Disable Disable false Disable Disable Auto Not Applicable N/A
3/2 Enable Enable Disable false Disable Disable Auto Not Applicable N/A
3/3 Enable Enable Disable false Disable Disable Auto Not Applicable N/A
3/4 Enable Enable Disable false Disable Disable Auto Not Applicable N/A

On an additional question about LED’s, do you know if there is a command to stop them all flashing?

Have a couple of 8404’s out the box where two all ports are flashing, other two are not and have no idea why?

Many thanks

Userlevel 2

I have checked the “Operrouting” on my side. All UNI ports with a VLAN and an IP address attached to this VLAN are on Operrouting enabled. The NNI ports with only the BB VLANs 4051-4052 are on Operrouting disabled. This makes sense. The doc says: Shows the routing status of the port.

 

The only thing I know about the flashing port LEDs on VOSS is when they are administratively shut. By default they are all shut.

 

Do you have configured the SMLT VLANs with the same I-SID on both cluster members so they are connected via the virtual IST? (show mlt i-sid)

Maybe there is a problem on L1 with the SFP? Do you have something in the DDI informations? (show pluggable-optical-modules detail)

 

 

Userlevel 6
Badge

Hi Dany,

Thanks for putting the time in to look at it, much appreciated.

Here is the output of the port with the issue and one that is OK, so look fairly the same:


====================================================================================================
Pluggable Optical Module Info 3/1 Detail
====================================================================================================
Port: 3/1
Type: GbicSx
DDM Supported : TRUE
Vendor Name : AVAGO Partnumber : AFBR-5715PZ SKU:
Vendor REV : Vendor SN : AM1152SXE9D
Vendor Date : 12/29/11
Wavelength : 850.00 nm

Digital Diagnostic Interface Supported

Optics Status : Ok
Calibration : Internal
RX Power Measurement : Average
Auxiliary 1 Monitoring : Not Implemented
Auxiliary 2 Monitoring : Not Implemented

----------------------------------------------------------------------------------------------------
LOW_ALARM LOW_WARN ACTUAL HIGH_WARN HIGH_ALARM THRESHOLD
THRESHOLD THRESHOLD VALUE THRESHOLD THRESHOLD STATUS
----------------------------------------------------------------------------------------------------
Temp(C) -40.0 -10.0 32.1875 85.0 100.0 Normal
Voltage(V) 2.9700 3.1349 3.2877 3.4650 3.6300 Normal
Bias(mA) 2.0 2.0 6.1580 10.0 12.0 Normal
TxPower(dBm) -12.3000 -10.8000 -5.5000 0.0 0.4000 Normal
RxPower(dBm) -40.0 -17.0 -6.1000 0.0 2.5000 Normal

Extreme Extended Diagnostic Information
----------------------------------------------------------------------------------------------------
Not Available

====================================================================================================
Pluggable Optical Module Info 3/4 Detail
====================================================================================================
Port: 3/4
Type: GbicSx
DDM Supported : TRUE
Vendor Name : AVAGO Partnumber : AFBR-5715PZ SKU:
Vendor REV : Vendor SN : AM1152SXBY9
Vendor Date : 12/28/11
Wavelength : 850.00 nm

Digital Diagnostic Interface Supported

Optics Status : Ok
Calibration : Internal
RX Power Measurement : Average
Auxiliary 1 Monitoring : Not Implemented
Auxiliary 2 Monitoring : Not Implemented

----------------------------------------------------------------------------------------------------
LOW_ALARM LOW_WARN ACTUAL HIGH_WARN HIGH_ALARM THRESHOLD
THRESHOLD THRESHOLD VALUE THRESHOLD THRESHOLD STATUS
----------------------------------------------------------------------------------------------------
Temp(C) -40.0 -10.0 34.7773 85.0 100.0 Normal
Voltage(V) 2.9700 3.1349 3.2813 3.4650 3.6300 Normal
Bias(mA) 2.0 2.0 6.4640 10.0 12.0 Normal
TxPower(dBm) -12.3000 -10.8000 -5.4000 0.0 0.4000 Normal
RxPower(dBm) -40.0 -17.0 -6.7000 0.0 2.5000 Normal

Extreme Extended Diagnostic Information
----------------------------------------------------------------------------------------------------
Not Available
====================================================================================================

Here is the show mlt i-sid. I used an MLT that match the port i.e. 301 is port 3/1, 304 is 3/4, 423 is 4/23 and so on

DC1-Core2:1#% show mlt i-sid
************************************************************************************
Command Execution Time: Fri Feb 12 09:03:19 2021 GMT
************************************************************************************
==============================================================================================================
MLT Isid Info
==============================================================================================================
ISID ISID ISID
MLTID IFINDEX ID VLANID C-VID TYPE ORIGIN NAME BPDU
--------------------------------------------------------------------------------------------------------------
301 6444 12110626 626 N/A CVLAN CONFIG ISID-12110626
301 6444 12110627 627 N/A CVLAN CONFIG ISID-12110627
301 6444 12110935 935 N/A CVLAN CONFIG ISID-12110935
301 6444 12111003 1003 N/A CVLAN CONFIG ISID-12111003
301 6444 12112002 2002 N/A CVLAN CONFIG ISID-12112002
302 6445 12110627 627 N/A CVLAN CONFIG ISID-12110627
302 6445 12110827 827 N/A CVLAN CONFIG ISID-12110827
302 6445 12110927 927 N/A CVLAN CONFIG ISID-12110927
302 6445 12111003 1003 N/A CVLAN CONFIG ISID-12111003
302 6445 12112002 2002 N/A CVLAN CONFIG ISID-12112002
303 6446 12110631 631 N/A CVLAN CONFIG ISID-12110631
303 6446 12110828 828 N/A CVLAN CONFIG ISID-12110828
303 6446 12110831 831 N/A CVLAN CONFIG ISID-12110831
303 6446 12110931 931 N/A CVLAN CONFIG ISID-12110931
303 6446 12111003 1003 N/A CVLAN CONFIG ISID-12111003
303 6446 12112002 2002 N/A CVLAN CONFIG ISID-12112002
304 6447 12110632 632 N/A CVLAN CONFIG ISID-12110632
304 6447 12110832 832 N/A CVLAN CONFIG ISID-12110832
304 6447 12110932 932 N/A CVLAN CONFIG ISID-12110932
304 6447 12111003 1003 N/A CVLAN CONFIG ISID-12111003
304 6447 12111100 1100 N/A CVLAN CONFIG ISID-12111100
304 6447 12112002 2002 N/A CVLAN CONFIG ISID-12112002

Each of the VLANs used have IP address associated, except for VLAN 2002, but this is being used on other ports without an issue:

DC1-Core2:1#% show ip interface || 626,627,935,2002
************************************************************************************
Command Execution Time: Fri Feb 12 09:06:37 2021 GMT
************************************************************************************
====================================================================================================
IP Interface - GlobalRouter
====================================================================================================
INTERFACE IP NET BCASTADDR REASM VLAN BROUTER IPSEC IP
ADDRESS MASK FORMAT MAXSIZE ID PORT STATE NAME
----------------------------------------------------------------------------------------------------
Vlan626 192.168.161.146 255.255.255.240 ones 1500 626 false disable
Vlan627 192.168.161.162 255.255.255.240 ones 1500 627 false disable
Vlan935 192.168.102.34 255.255.255.240 ones 1500 935 false disable

Strange right, might open a GTAC case as doesn’t seem straight forward. To me it seems like a L1 issue, primarily as all the other ports are OK and config wise I can’t see any difference.

If you have anymore ideas let me know, if I find out anything in the meantime I’ll post back.

Many thanks

Userlevel 2

You can try to move the SFP to another slot and or replace the SFP. 

I found another article and it looks very similar to your case:

https://extremeportal.force.com/ExtrArticleDetail?an=000060397&q=amber

Userlevel 6
Badge

Issue turned out to be with the AVAGO Gbic.

So although the same where used in other slots, had been swapped with another Gbic that seemed to working in another slot and equally had been moved working from an S8 to VOSS, something about it just wasn’t compatible.

These GBIC’s turned out not to be on the Extreme approved list, and thereby replacing with Extreme badged it worked fine.

Raised with GTAC whom identified the issue and provided this link to identify supported GBICs

https://optics.extremenetworks.com/