LACP issues

  • 0
  • 1
  • Problem
  • Updated 1 year ago
  • Solved
I have some 450G2 switches that I can not get LACP LAG to work on.  Core 670 switch is grouping ports 19 and 20.  Sh lacp lag displays:
Member     Port      Rx           Sel          Mux            Actor     Partner
Port       Priority  State        Logic        State          Flags     Port
--------------------------------------------------------------------------------
19         0         Current      Selected     Collect-Dist   A-GSCD--  1049
20         0         Current      Selected     Attached       A-GS----  2049
================================================================================
Actor Flags: A-Activity, T-Timeout, G-Aggregation, S-Synchronization
             C-Collecting, D-Distributing, F-Defaulted, E-Expired

Remote switch shows:
Member     Port      Rx           Sel          Mux            Actor     Partner
Port       Priority  State        Logic        State          Flags     Port
--------------------------------------------------------------------------------
1:49       0         Current      Selected     Collect-Dist   A-GSCD--  1019
2:49       0         Defaulted    Unselected   Detached       A-G---F-  0
================================================================================
Actor Flags: A-Activity, T-Timeout, G-Aggregation, S-Synchronization
             C-Collecting, D-Distributing, F-Defaulted, E-Expired

I have tried moving the fiber cable to different pairs.  removed and added the grouping.  Disabled ports, renabled ports, all with no affect.  Link lights are on both switches.


What can be wrong?

Not sure if this is a connected issue or not, but all of my network switches are also showing a loop detected on the uplink port from the core.  ELRP is active on edge switches with the uplink ports excluded.
<Warn:ELRP.Report.Message> Slot-1: [CLI:vlan:2] LOOP DETECTED : 846382 transmitted, 9114521 received, ingress slot:port (1:49) egress slot:port (1:49)

Fairly new to Extreme switching so bear with me.
Photo of Jason Hilt

Jason Hilt

  • 400 Points 250 badge 2x thumb

Posted 2 years ago

  • 0
  • 1
Photo of Patrick Voss

Patrick Voss, Alum

  • 11,574 Points 10k badge 2x thumb
Hello Jason,

What does the "show sharing" display? If LACP wasn't working correctly then it should show the port as active but not added into the aggregator so it will display a "-" instead of a "y". Does it link up when you configure it without LACP? This could be a good test.
Photo of Ty Kolff

Ty Kolff

  • 1,098 Points 1k badge 2x thumb
What version of EXOS are you using on the 450-G2's?
Photo of Jason Hilt

Jason Hilt

  • 400 Points 250 badge 2x thumb
21.1.1.4 Patch1-2.

sharing shows AGG Mbr -

Can't check without LACP right now while school is in session. 
Photo of Patrick Voss

Patrick Voss, Alum

  • 11,574 Points 10k badge 2x thumb
Hello Jason,

So the port is active and has a "y" next to both ports? on both sides?
Photo of Jason Hilt

Jason Hilt

  • 400 Points 250 badge 2x thumb
There is only a Y on one port on each end.  The other port shows a - for the Agg Mbr.
Core:
    19     19     LACP       1    L2        A     19        Y     A       8
                                            L2                20        -     A      14
Remote switch:
  1:49   1:49     LACP       1    L2        A     1:49      Y     A       3
                                              L2                2:49      -     A      12
================================================================================


Sh lacp lag on the core is:
Member     Port      Rx           Sel          Mux            Actor     Partner
Port       Priority  State        Logic        State          Flags     Port
--------------------------------------------------------------------------------
19         0         Current      Selected     Collect-Dist   A-GSCD--  1049
20         0         Current      Selected     Attached       A-GS----  2049
================================================================================

Sh lacp lag on the remote switch is:
Member     Port      Rx           Sel          Mux            Actor     Partner
Port       Priority  State        Logic        State          Flags     Port
--------------------------------------------------------------------------------
1:49       0         Current      Selected     Collect-Dist   A-GSCD--  1019
2:49       0         Defaulted    Unselected   Detached       A-G---F-  0
================================================================================

Connections are 10gig fiber on SM fiber using Extreme Network branded SumitomoElectric SFP+ optics.
Photo of Patrick Voss

Patrick Voss, Alum

  • 11,574 Points 10k badge 2x thumb
Hello Jason,

The "A" indicates the port is active. When there is a "Y" it means that it is active in the share group. So in this case port 20 and 2:49 are not passing traffic. The ELRP messages may be unrelated to the LACP issue. I would look into the other side of 1:49 and see if ELRP is being triggered on that switch.
Photo of Jason Hilt

Jason Hilt

  • 400 Points 250 badge 2x thumb
ELRP is not configured on the core switch.  Only on edge switches.
Photo of Ty Kolff

Ty Kolff

  • 1,098 Points 1k badge 2x thumb
We discovered an issue with LACP last week with EXOS version 21.1.2.  It's a known issue that will be resolved in 21.1.3, but it hasn't been released yet.  In this case, we had to roll back to 16.1.4.2 patch 1-7.  
Photo of Jason Hilt

Jason Hilt

  • 400 Points 250 badge 2x thumb
Is this a random issue as we have quite a few other buildings with the same switches, SFPs, and cables that are working just fine running 21.1.1.4?
Photo of Ty Kolff

Ty Kolff

  • 1,098 Points 1k badge 2x thumb
I have others running just fine on 21.1.1 as well as 21.1.2, but we discovered this issue while doing bench testing before putting these switches into production.  
Photo of Erik Auerswald

Erik Auerswald, Embassador

  • 12,960 Points 10k badge 2x thumb
Hi Jason,

did you verify the cabling with e.g. "show edp port 19,20" resp. "show edp port 1:49,2:49"?

You can use the command "show lacp member-port <NUMBER> detail" for more info regarding LACP on the ports.

Erik
Photo of Jason Hilt

Jason Hilt

  • 400 Points 250 badge 2x thumb
sh edp port 1:49,2:49 shows correct remote ports on the core.
Port   Neighbor                       Neighbor-ID         Remote  Age    Num
                                                                                 Port         Vlans
=============================================================================
1:49   FC-Core-670               00:00:00:04:96:9b:37:50   1:12   9      15
2:49   FC-Core-670               00:00:00:04:96:9b:37:50   1:11   5      15
=============================================================================

sh lacp lag 1:49    after a disable/enable on port 2:49.  RX state went to expired before going to Defaulted.

Member     Port      Rx           Sel          Mux            Actor     Partner
Port       Priority  State        Logic        State          Flags     Port
--------------------------------------------------------------------------------
1:49       0         Current      Selected     Collect-Dist   A-GSCD--  1012
2:49       0         Defaulted    Unselected   Detached       A-G---F-  0
================================================================================

Going to try reseating the optics...again and maybe a different fiber pair.
Photo of Erik Auerswald

Erik Auerswald, Embassador

  • 12,960 Points 10k badge 2x thumb
Hi Jason,

with "show lacp member-port 1:49 detail" you get counters regarding sent and received LACP PDUs, and the LACP system IDs.

I would expect that port 2:49 does not receive LACP PDUs.

Erik
(Edited)
Photo of Jason Hilt

Jason Hilt

  • 400 Points 250 badge 2x thumb
So here is the trouble port after a SFP reseat, fiber pair change, and disable/enable.

Member     Port      Rx           Sel          Mux            Actor     Partner
Port       Priority  State        Logic        State          Flags     Port
--------------------------------------------------------------------------------
2:49       0         Defaulted    Unselected   Detached       A-G---F-  0
Up            : Yes
Enabled       : Yes
Link State    : Up
Actor Churn   : True
Partner Churn : True
Ready_N       : No
Wait pending  : No
Ack pending   : No
LAG Id:
  S.pri:0   , S.id:00:00:00:00:00:00, K:0x0000, P.pri:0   , P.num:0
  T.pri:0   , T.id:00:00:00:00:00:00, L:0x0000, Q.pri:0   , Q.num:0
Stats:
  Rx - Accepted                                      : 0
  Rx - Dropped due to error in verifying PDU         : 0
  Rx - Dropped due to LACP not being up on this port : 0
  Rx - Dropped due to matching own MAC               : 0

  Tx - Sent successfully                             : 128
  Tx - Transmit error                                : 0
================================================================================
Actor Flags: A-Activity, T-Timeout, G-Aggregation, S-Synchronization
             C-Collecting, D-Distributing, F-Defaulted, E-Expired

Here is the core side of the link

Member     Port      Rx           Sel          Mux            Actor     Partner
Port       Priority  State        Logic        State          Flags     Port
--------------------------------------------------------------------------------
11         0         Current      Selected     Attached       A-GS----  2049
Up            : Yes
Enabled       : Yes
Link State    : Up
Actor Churn   : False
Partner Churn : True
Ready_N       : Yes
Wait pending  : No
Ack pending   : No
LAG Id:
  S.pri:0   , S.id:00:04:96:9b:37:50, K:0x03f3, P.pri:0   , P.num:1011
  T.pri:0   , T.id:02:04:96:9b:02:c4, L:0x0419, Q.pri:0   , Q.num:2049
Stats:
  Rx - Accepted                                      : 268
  Rx - Dropped due to error in verifying PDU         : 0
  Rx - Dropped due to LACP not being up on this port : 0
  Rx - Dropped due to matching own MAC               : 0

  Tx - Sent successfully                             : 13
  Tx - Transmit error                                : 0
================================================================================
Actor Flags: A-Activity, T-Timeout, G-Aggregation, S-Synchronization
             C-Collecting, D-Distributing, F-Defaulted, E-Expired


Going to try a different SFP next.
Photo of Erik Auerswald

Erik Auerswald, Embassador

  • 12,960 Points 10k badge 2x thumb
So it seems the LACP PDUs from the core switch do not reach the X450, but the other direction works.

Did you verify that the transmission fiber strand from the X450 port 2:49 actually connects to the correct X670 port's receive fiber strand, e.g. by using EDP on the X670 switch?

Did you check the SFP types on both sides (core and access)?

Do you see errors on the ports (e.g. "show port 2:49 rxerrors no-refresh")?
Photo of Jason Hilt

Jason Hilt

  • 400 Points 250 badge 2x thumb
Ports are correct.  Disabled remote port 2:49 and port 11 on the core went offline.
SFP info is the same on both ends.
No txerrors or rxerrors on either port.
Changed out SFP with a working connection on both ends.  No luck.  Changed out fiber on both ends.  No luck.  Changed fiber pair again - third time. No luck.
Photo of Erik Auerswald

Erik Auerswald, Embassador

  • 12,960 Points 10k badge 2x thumb
Hm, in the first post you showed ports 19&20 on the core, but EDP showed ports 11&12 on the core. Do you have configured the correct core ports for port sharing with LACP?
Photo of Jason Hilt

Jason Hilt

  • 400 Points 250 badge 2x thumb
Sorry about that.  My initial problem was on a different network closet which fixed itself after I rebooted all of the switches that were having this problem.  The latest posts deal with the one remaining closet that still isn't working.
Photo of Erik Auerswald

Erik Auerswald, Embassador

  • 12,960 Points 10k badge 2x thumb
I would suggest opening a case with GTAC, since the configuration looks OK. LACP PDUs are sent on one end, but not received (or registered / correctly interpreted) on the other.
Photo of Jason Hilt

Jason Hilt

  • 400 Points 250 badge 2x thumb
Yep, I think GTAC is going to be needed.  Quite a few of the switches that were working correctly have now gone to a defaulted state on one of the links.  Very frustrating.  Hopefully GTAC does better on this issue than the last one they couldn't figure out.
Photo of Drew C.

Drew C., Community Manager

  • 38,566 Points 20k badge 2x thumb
Hi Jason,
Was GTAC able to help you get LACP working as expected? I didn't see a case in the system in your name, so I couldn't tell.
Photo of Jason Hilt

Jason Hilt

  • 400 Points 250 badge 2x thumb
Yes and no.  They think it's a bug in the software but I didn't hear back for certain that it was.  I ended up rebooting the stacks and both links joined up.
I just had the same issue on 3 other stacks at another building when I had to take down the network because of a planned power outage.  I just rebooted them, some 3 times, and the links joined up finally.
I'll look at upgrading the ExtremeXOS this summer if a fix or new version is out.
Photo of Drew C.

Drew C., Community Manager

  • 38,566 Points 20k badge 2x thumb
Very odd. Glad its working now though.
I'm going to mark this one as Solved. If you get a definitive answer, hopefully you'll remember the thread and will come back to update it.