link down - Local Fault

  • 0
  • 1
  • Problem
  • Updated 2 years ago
  • Solved
I have been seeing an issue in my virtual environment of randomly losing connectivity to iSCSI LUNs.  My hosts are showing a loss of connectivity, I looked at the logs on my X670V-48t and i found what looked to be the same time frame of the port going down stating :
05/22/2016 19:23:28.19 <Info:vlan.msgs.portLinkStateDown> Slot-1: Port 1:21 link down - Local fault
05/22/2016 19:23:57.64 <Info:vlan.msgs.portLinkStateUp> Slot-1: Port 1:21 link UP at speed 10 Gbps and full-duplex
05/22/2016 19:23:58.99 <Warn:lldp.DCBX.CfgMisMatchDtect> Slot-1: Configuration mismatch detected by DCBX (Baseline v1.01) for the PFC TLV on port 1:21.
05/22/2016 19:23:59.99 <Noti:lldp.DCBX.CfgMisMatchRslv> Slot-1: Configuration mismatch resolved by DCBX (Baseline v1.01) for the PFC TLV on port 1:21.

I need a little insight on what is possibly going on here.
Photo of Jason Weems

Jason Weems

  • 412 Points 250 badge 2x thumb

Posted 2 years ago

  • 0
  • 1
Photo of Taykin Izzet

Taykin Izzet , Employee

  • 3,106 Points 3k badge 2x thumb
Jason,

If we look at the first message we see that the link went down and was detected on the local port. The port detects the link fault on its TX or RX pair based on IEEE 802.3ae-2002.

The final message is based on LLDP reporting that the configuration mismatch has been resolved (once the port comes back up). The configuration now matches, and the DCB feature should be operating properly.

I would focus on ruling out any layer1 issues such as cabling and port at this time. The port appears to go down briefly and then comes back up.
(Edited)
Photo of Jason Weems

Jason Weems

  • 412 Points 250 badge 2x thumb
Sorry but " The port detects the link fault on its TX or RX pair based on IEEE 802.3ae-2002." is a pretty vague statement, especially when I don't have access to view the standard as they want it to be bought for over $1000.  How about a little more clarification on that.  What defines a fault, for this and how is it defined local ?  
I understand the concept of the remote as the remote device sends that signal, but how does it know that the fault is local.  
This happened several times several minutes apart.  I am not currently seeing this issue the cabling has not changed or been touched.  I was not seeing this issue before as it seems to just appeared.  It has also happened on some other ports as well, so i don't really think that I have had a bunch of cables just go bad, 1 cable ok but 10, I don't believe that.  
Do you have another suggestion of a possible direction to look at to troubleshoot.
Photo of Taykin Izzet

Taykin Izzet , Employee

  • 3,106 Points 3k badge 2x thumb
So it appears that the issue is happening on multiple ports. Could you provide the following information:

1. It is the same ports affected each time?
2. Are the ports part of a Link Aggregation Group?
3. How frequent the issue is occurring?
4. The version of EXOS installed

More information on the fault detection process can be found in the following article:

https://gtacknowledge.extremenetworks.com/articles/Q_A/what-is-the-difference-between-local-fault-an...

Isolating the issue and ruling out any layer 1 issue is generally a first step in troubleshooting. If multiple ports are affected, please isolate the issue by moving one of the problematic links to another port to see whether the problem follows. Also check for any errors on the port using the command show port <number> rx errors and show port <number> tx errors.
Photo of Kevin Kim

Kevin Kim, Employee

  • 2,266 Points 2k badge 2x thumb
- Local Fault indicates loss of signal detected on the receive data path of a local port
- Remote Fault indicates a fault on the transmit path

(Edited)
Photo of Jason Weems

Jason Weems

  • 412 Points 250 badge 2x thumb
The ports affected are the same set of ports each time.  
None of the ports are part of an aggregation.  
It happened to each of the ports 3 times in a 2hr period.
XOS version is 15.3.3.5

This was happening to 9 different ports which were attached to 3 different hosts, 3 ports per Host.
Host A - 1:3 mgmt, 2:26 iSCSI, 2:18 VM data (mulitple vlans)
Host B - 1:38 mgmt, 1:21 iSCSI, 2:8 VM data (multiple vlans)
Host C - 2:21 mgmt, 2:14 iSCSI, 1:12 VM data (multiple vlans)

2 of the host, Host A and Host B were affected enough that they needed to be rebooted.  Host C for some reason did not show any signs of distress.

Here is the full log from the time frame:
05/22/2016 22:20:35.65 <Info:vlan.msgs.portLinkStateUp> Slot-1: Port 1:38 link UP at speed 1 Gbps and full-duplex05/22/2016 22:20:31.49 <Info:vlan.msgs.portLinkStateDown> Slot-1: Port 1:38 link down
05/22/2016 22:20:24.44 <Info:vlan.msgs.portLinkStateUp> Slot-1: Port 1:38 link UP at speed 1 Gbps and full-duplex
05/22/2016 22:20:20.44 <Info:vlan.msgs.portLinkStateDown> Slot-1: Port 1:38 link down
05/22/2016 22:15:48.62 <Noti:lldp.DCBX.CfgMisMatchRslv> Slot-1: Configuration mismatch resolved by DCBX (Baseline v1.01) for the PFC TLV on port 1:21.
05/22/2016 22:15:47.62 <Warn:lldp.DCBX.CfgMisMatchDtect> Slot-1: Configuration mismatch detected by DCBX (Baseline v1.01) for the PFC TLV on port 1:21.
05/22/2016 22:15:46.27 <Info:vlan.msgs.portLinkStateUp> Slot-1: Port 1:21 link UP at speed 10 Gbps and full-duplex
05/22/2016 22:15:38.36 <Info:vlan.msgs.portLinkStateDown> Slot-1: Port 1:21 link down - Local fault
05/22/2016 22:15:38.21 <Info:vlan.msgs.portLinkStateUp> Slot-1: Port 1:21 link UP at speed 10 Gbps and full-duplex
05/22/2016 22:15:38.19 <Info:vlan.msgs.portLinkStateUp> Slot-1: Port 2:8 link UP at speed 10 Gbps and full-duplex
05/22/2016 22:15:36.06 <Info:vlan.msgs.portLinkStateUp> Slot-1: Port 1:38 link UP at speed 1 Gbps and full-duplex
05/22/2016 22:15:31.39 <Info:vlan.msgs.portLinkStateDown> Slot-1: Port 1:38 link down
05/22/2016 22:15:30.80 <Info:vlan.msgs.portLinkStateUp> Slot-1: Port 1:38 link UP at speed 1 Gbps and full-duplex
05/22/2016 22:15:26.60 <Info:vlan.msgs.portLinkStateDown> Slot-1: Port 1:21 link down - Local fault
05/22/2016 22:15:26.58 <Info:vlan.msgs.portLinkStateDown> Slot-1: Port 2:8 link down - Local fault
05/22/2016 22:15:26.42 <Info:vlan.msgs.portLinkStateDown> Slot-1: Port 1:38 link down
05/22/2016 22:15:24.86 <Info:vlan.msgs.portLinkStateUp> Slot-1: Port 1:38 link UP at speed 1 Gbps and full-duplex
05/22/2016 22:15:20.21 <Info:vlan.msgs.portLinkStateDown> Slot-1: Port 1:38 link down
05/22/2016 21:43:30.60 <Info:vlan.msgs.portLinkStateUp> Slot-1: Port 1:38 link UP at speed 1 Gbps and full-duplex
05/22/2016 21:43:26.59 <Info:vlan.msgs.portLinkStateDown> Slot-1: Port 1:38 link down
05/22/2016 21:43:19.36 <Info:vlan.msgs.portLinkStateUp> Slot-1: Port 1:38 link UP at speed 1 Gbps and full-duplex
05/22/2016 21:43:15.13 <Info:vlan.msgs.portLinkStateDown> Slot-1: Port 1:38 link down
05/22/2016 21:40:46.14 <Info:vlan.msgs.portLinkStateUp> Slot-1: Port 1:3 link UP at speed 1 Gbps and full-duplex
05/22/2016 21:40:42.03 <Info:vlan.msgs.portLinkStateDown> Slot-1: Port 1:3 link down
05/22/2016 21:38:52.46 <Noti:lldp.DCBX.CfgMisMatchRslv> Slot-1: Configuration mismatch resolved by DCBX (Baseline v1.01) for the PFC TLV on port 1:21.
05/22/2016 21:38:51.46 <Warn:lldp.DCBX.CfgMisMatchDtect> Slot-1: Configuration mismatch detected by DCBX (Baseline v1.01) for the PFC TLV on port 1:21.
05/22/2016 21:38:50.20 <Info:vlan.msgs.portLinkStateUp> Slot-1: Port 1:21 link UP at speed 10 Gbps and full-duplex
05/22/2016 21:38:42.35 <Info:vlan.msgs.portLinkStateDown> Slot-1: Port 1:21 link down - Local fault
05/22/2016 21:38:42.29 <Info:vlan.msgs.portLinkStateUp> Slot-1: Port 2:8 link UP at speed 10 Gbps and full-duplex
05/22/2016 21:38:42.20 <Info:vlan.msgs.portLinkStateUp> Slot-1: Port 1:21 link UP at speed 10 Gbps and full-duplex
05/22/2016 21:38:29.85 <Info:vlan.msgs.portLinkStateUp> Slot-1: Port 1:38 link UP at speed 1 Gbps and full-duplex
05/22/2016 21:38:25.24 <Info:vlan.msgs.portLinkStateDown> Slot-1: Port 1:38 link down
05/22/2016 21:38:23.51 <Info:vlan.msgs.portLinkStateUp> Slot-1: Port 1:38 link UP at speed 1 Gbps and full-duplex
05/22/2016 21:38:17.02 <Info:vlan.msgs.portLinkStateDown> Slot-1: Port 1:38 link down
05/22/2016 21:36:28.00 <Info:vlan.msgs.portLinkStateUp> Slot-1: Port 1:38 link UP at speed 100 Mbps and full-duplex
05/22/2016 21:36:24.12 <Info:vlan.msgs.portLinkStateDown> Slot-1: Port 2:8 link down - Local fault
05/22/2016 21:36:24.10 <Info:vlan.msgs.portLinkStateDown> Slot-1: Port 1:21 link down - Local fault
05/22/2016 21:36:23.97 <Info:vlan.msgs.portLinkStateDown> Slot-1: Port 1:38 link down
05/22/2016 21:32:38.52 <Noti:lldp.DCBX.CfgMisMatchRslv> Slot-1: Configuration mismatch resolved by DCBX (Baseline v1.01) for the PFC TLV on port 2:26.
05/22/2016 21:32:37.52 <Warn:lldp.DCBX.CfgMisMatchDtect> Slot-1: Configuration mismatch detected by DCBX (Baseline v1.01) for the PFC TLV on port 2:26.
05/22/2016 21:32:36.21 <Info:vlan.msgs.portLinkStateUp> Slot-1: Port 2:26 link UP at speed 10 Gbps and full-duplex
05/22/2016 21:32:36.20 <Info:vlan.msgs.portLinkStateUp> Slot-1: Port 2:18 link UP at speed 10 Gbps and full-duplex
05/22/2016 21:32:33.72 <Info:vlan.msgs.portLinkStateUp> Slot-1: Port 1:3 link UP at speed 1 Gbps and full-duplex
05/22/2016 21:32:29.28 <Info:vlan.msgs.portLinkStateDown> Slot-1: Port 1:3 link down
05/22/2016 21:32:27.03 <Info:vlan.msgs.portLinkStateUp> Slot-1: Port 1:3 link UP at speed 1 Gbps and full-duplex
05/22/2016 21:32:24.19 <Info:vlan.msgs.portLinkStateDown> Slot-1: Port 2:26 link down - Local fault
05/22/2016 21:32:24.18 <Info:vlan.msgs.portLinkStateDown> Slot-1: Port 2:18 link down - Local fault
05/22/2016 21:32:18.52 <Info:vlan.msgs.portLinkStateDown> Slot-1: Port 1:3 link down
05/22/2016 20:47:03.66 <Info:vlan.msgs.portLinkStateUp> Slot-1: Port 2:21 link UP at speed 1 Gbps and full-duplex
05/22/2016 20:47:00.41 <Info:vlan.msgs.portLinkStateUp> Slot-1: Port 2:14 link UP at speed 10 Gbps and full-duplex
05/22/2016 20:46:59.54 <Info:vlan.msgs.portLinkStateDown> Slot-1: Port 2:21 link down
05/22/2016 20:46:59.09 <Info:vlan.msgs.portLinkStateUp> Slot-1: Port 2:21 link UP at speed 1 Gbps and full-duplex
05/22/2016 20:46:55.02 <Info:vlan.msgs.portLinkStateDown> Slot-1: Port 2:21 link down
05/22/2016 20:46:53.29 <Info:vlan.msgs.portLinkStateUp> Slot-1: Port 1:12 link UP at speed 10 Gbps and full-duplex
05/22/2016 20:46:47.54 <Info:vlan.msgs.portLinkStateDown> Slot-1: Port 1:12 link down - Local fault
05/22/2016 20:46:46.72 <Info:vlan.msgs.portLinkStateUp> Slot-1: Port 1:12 link UP at speed 10 Gbps and full-duplex
05/22/2016 20:46:46.61 <Info:vlan.msgs.portLinkStateDown> Slot-1: Port 1:12 link down
05/22/2016 20:46:02.06 <Info:vlan.msgs.portLinkStateDown> Slot-1: Port 2:14 link down - Local fault
05/22/2016 20:43:07.41 <Info:vlan.msgs.portLinkStateUp> Slot-1: Port 2:14 link UP at speed 10 Gbps and full-duplex
05/22/2016 20:43:07.30 <Info:vlan.msgs.portLinkStateDown> Slot-1: Port 2:14 link down
05/22/2016 20:43:07.21 <Info:vlan.msgs.portLinkStateUp> Slot-1: Port 2:14 link UP at speed 10 Gbps and full-duplex
05/22/2016 20:42:26.64 <Info:vlan.msgs.portLinkStateUp> Slot-1: Port 1:12 link UP at speed 10 Gbps and full-duplex
05/22/2016 20:42:20.88 <Info:vlan.msgs.portLinkStateDown> Slot-1: Port 1:12 link down - Local fault
05/22/2016 20:42:20.04 <Info:vlan.msgs.portLinkStateUp> Slot-1: Port 1:12 link UP at speed 10 Gbps and full-duplex
05/22/2016 20:42:19.96 <Info:vlan.msgs.portLinkStateDown> Slot-1: Port 1:12 link down
05/22/2016 20:40:35.22 <Info:vlan.msgs.portLinkStateUp> Slot-1: Port 1:12 link UP at speed 10 Gbps and full-duplex
05/22/2016 20:40:33.34 <Info:vlan.msgs.portLinkStateUp> Slot-1: Port 2:21 link UP at speed 1 Gbps and full-duplex
05/22/2016 20:40:29.71 <Info:vlan.msgs.portLinkStateDown> Slot-1: Port 1:12 link down - Local fault
05/22/2016 20:40:28.75 <Info:vlan.msgs.portLinkStateDown> Slot-1: Port 2:21 link down
05/22/2016 20:40:28.07 <Info:vlan.msgs.portLinkStateDown> Slot-1: Port 2:14 link down - Local fault
05/22/2016 20:09:33.98 <Noti:lldp.DCBX.MultiPeerRslv> Slot-1: Multiple LLDP peers resolved by DCBX (Baseline v1.01) on port 1:21.
05/22/2016 20:07:04.01 <Warn:lldp.DCBX.MultiPeerDtect> Slot-1: Multiple(2) LLDP peers detected by DCBX (Baseline v1.01) on port 1:21.
05/22/2016 19:28:50.00 <Info:vlan.msgs.portLinkStateUp> Slot-1: Port 1:38 link UP at speed 1 Gbps and full-duplex
05/22/2016 19:28:46.02 <Info:vlan.msgs.portLinkStateDown> Slot-1: Port 1:38 link down
05/22/2016 19:28:39.18 <Info:vlan.msgs.portLinkStateUp> Slot-1: Port 1:38 link UP at speed 1 Gbps and full-duplex
05/22/2016 19:28:35.12 <Info:vlan.msgs.portLinkStateDown> Slot-1: Port 1:38 link down
05/22/2016 19:23:59.99 <Noti:lldp.DCBX.CfgMisMatchRslv> Slot-1: Configuration mismatch resolved by DCBX (Baseline v1.01) for the PFC TLV on port 1:21.
05/22/2016 19:23:58.99 <Warn:lldp.DCBX.CfgMisMatchDtect> Slot-1: Configuration mismatch detected by DCBX (Baseline v1.01) for the PFC TLV on port 1:21.
05/22/2016 19:23:57.64 <Info:vlan.msgs.portLinkStateUp> Slot-1: Port 1:21 link UP at speed 10 Gbps and full-duplex
05/22/2016 19:23:57.56 <Info:vlan.msgs.portLinkStateUp> Slot-1: Port 2:8 link UP at speed 10 Gbps and full-duplex
05/22/2016 19:23:45.39 <Info:vlan.msgs.portLinkStateUp> Slot-1: Port 1:38 link UP at speed 1 Gbps and full-duplex
05/22/2016 19:23:40.98 <Info:vlan.msgs.portLinkStateDown> Slot-1: Port 1:38 link down
05/22/2016 19:23:38.77 <Info:vlan.msgs.portLinkStateUp> Slot-1: Port 1:38 link UP at speed 1 Gbps and full-duplex
05/22/2016 19:23:32.41 <Info:vlan.msgs.portLinkStateDown> Slot-1: Port 1:38 link down
05/22/2016 19:23:31.84 <Info:vlan.msgs.portLinkStateUp> Slot-1: Port 1:38 link UP at speed 100 Mbps and full-duplex
05/22/2016 19:23:28.27 <Info:vlan.msgs.portLinkStateDown> Slot-1: Port 2:8 link down - Local fault
05/22/2016 19:23:28.19 <Info:vlan.msgs.portLinkStateDown> Slot-1: Port 1:21 link down - Local fault
05/22/2016 19:23:27.69 <Info:vlan.msgs.portLinkStateDown> Slot-1: Port 1:38 link down
Photo of Drew C.

Drew C., Community Manager

  • 39,124 Points 20k badge 2x thumb
Jason, are you still having link troubles?
Photo of Jason Weems

Jason Weems

  • 412 Points 250 badge 2x thumb

I am still having trouble but I don't believe it is an issue with the extreme equipment.  I believe it is another issue.  Thank you.

Photo of Ty Kolff

Ty Kolff

  • 1,098 Points 1k badge 2x thumb
Just an FYI for anyone else who see this.  I just had this issue with a 10gb connection between a x620 and an x460.  I swapped out the SFP+ cable and the problem went away.