ExtremeSwitching (EXOS)

Expand all | Collapse all

Port Load sharing

  • 1.  Port Load sharing

    Posted 12-02-2014 00:57
    After i enable Lacp .. port is not sharing. Utilization is high on Link 1 only.
    I want to use load balance on both links.
    Please kindly see the below output and advice thanks.

    * I try L2 static, L3_L4 and LACP. all the same ports are not load balance. *

    enable sharing 1 grouping 1,4

    enable sharing 1 grouping 1,4 L3_L4

    enable sharing 1 grouping 1,4 algorithm address-based L3_L4 lacp


    CoreSW # sh port 1,4 utilization bandwithPort Link Link Rx Peak Rx Tx Peak Tx
    State Speed % bandwidth % bandwidth % bandwidth % bandwidth
    ================================================================================
    Link_1> A 1000 82.06 94.94 48.23 55.72
    Link_2> A 1000 0.25 0.30 1.34 1.57
    ================================================================================
    > indicates Port Display Name truncated past 8 characters
    Link State: A-Active, R-Ready, NP-Port Not Present, L-Loopback

    CoreSW #sh sharing
    Load Sharing Monitor
    Config Current Agg Ld Share Ld Share Agg Link Link Up
    Master Master Control Algorithm Group Mbr State Transitions
    ==============================================================================
    1 1 LACP L3_L4 1 Y A 0
    L3_L4 4 Y A 0
    ==============================================================================
    Link State: A-Active, D-Disabled, R-Ready, NP-Port not present, L-Loopback
    Load Sharing Algorithm: (L2) Layer 2 address based
    (L3_L4) Layer 3 address and Layer 4 port based
    Number of load sharing trunks: 1


    Core2 # sh lacp lag 1
    Lag Actor Actor Partner Partner Partner Agg Actor
    Sys-Pri Key MAC Sys-Pri Key Count MAC
    --------------------------------------------------------------------------------
    1 0 0x03e9 00:04:96:34:b2:e1 0 0x03e9 2 00:04:96:34:b2:e0

    Port list:

    Member Port Rx Sel Mux Actor Partner
    Port Priority State Logic State Flags Port
    --------------------------------------------------------------------------------
    1 0 Current Selected Collect-Dist A-GSCD-- 1001
    4 0 Current Selected Collect-Dist A-GSCD-- 1012
    ================================================================================
    Actor Flags: A-Activity, T-Timeout, G-Aggregation, S-Synchronization
    C-Collecting, D-Distributing, F-Defaulted, E-Expired


  • 2.  RE: Port Load sharing

    Posted 12-02-2014 06:01
    What model do you use in this example ? (Summit X250, 460, BD ? )

    LACP is between two extreme switches or server and 1 extreme switch ? (other vendor ?)

    What EXOS is on the switch ?

    --
    Jarek



  • 3.  RE: Port Load sharing

    Posted 12-02-2014 06:01
    Thanks Jarek,

    Core1 and Core 2 using X450a-24x

    LACP is between two Core Switches.
    ExtremeXOS version 15.3.1.4

    Please advice thanks.

    We want to use load balance on both links.


  • 4.  RE: Port Load sharing

    Posted 12-02-2014 06:01
    I have two X250e with config like bellow:
    "enable sharing 24 grouping 21-24 algorithm address-based L3_L4"

    and all is working well (EXOS ver 15.3.1.4-1-30).

    Did you enabled sharing on both swicthes?


  • 5.  RE: Port Load sharing

    Posted 12-02-2014 06:01
    could you please share the outcome of

    sh port 21,24 utilization bandwith


  • 6.  RE: Port Load sharing

    Posted 12-02-2014 06:01
    Yes i did sharing on both switches. Thanks.

    Please kindly share the show command of

    show port 21-24 utilization bandwidth

    Is it load balance on port 21-24 links?


  • 7.  RE: Port Load sharing

    Posted 12-02-2014 06:01
    Paul,

    you set those commands on switch like bellow:
    enable sharing 1 grouping 1,4
    enable sharing 1 grouping 1,4 L3_L4
    enable sharing 1 grouping 1,4 algorithm address-based L3_L4 lacp

    or only :

    enable sharing 1 grouping 1,4 algorithm address-based L3_L4 lacp

    I have on my switches:

    enable sharing 24 grouping 21-24 algorithm address-based L3_L4 lacp

    # sh ports 21-24 utilization bandwidth
    Port Link Link Rx Peak Rx Tx Peak Tx
    State Speed % bandwidth % bandwidth % bandwidth % bandwidth
    ================================================================================
    21 A 100 0.00 0.01 0.02 8.27
    22 A 100 1.61 1.61 45.28 45.28
    23 A 100 0.02 0.02 0.00 7.97
    24 A 100 1.54 2.96 43.32 90.63
    ================================================================================

    --
    Jarek



  • 8.  RE: Port Load sharing

    Posted 12-02-2014 06:01
    I use only one command same like you. But the both links are not load share as i mentioned above. Anyway how can i load balance on these port. Please advice


  • 9.  RE: Port Load sharing

    Posted 12-02-2014 06:01
    Link1 is 82 % and Link2 is only 0.25 % We want to achive load balanced on link1 and link2 Really appreciate your help and advice Jarek.


  • 10.  RE: Port Load sharing

    Posted 12-02-2014 06:01
    What exactly exos do you have 15.3.1.4-p1-xx <-?



  • 11.  RE: Port Load sharing

    Posted 12-02-2014 06:01
    Sorry my current active version is 15.3.3.5. 15.3.1.4 is on other partition. Thanks. Is it bug or firmware not supported? I do need your adivce on this. The links are in live production network.


  • 12.  RE: Port Load sharing

    Posted 12-02-2014 06:01
    Hmm, I don't use 15.3.3.5. Maybe there is a bug...
    Can you show me from both switches:
    sh sharing
    sh lacp
    sh lacp counters
    sh lacp lag 1 detail



  • 13.  RE: Port Load sharing

    Posted 12-02-2014 06:01
    I will provide it later. Currently i m outside. Thanks alot.


  • 14.  RE: Port Load sharing

    Posted 12-02-2014 06:01
    Core2 # sh sharing
    Load Sharing Monitor
    Config Current Agg Ld Share Ld Share Agg Link Link Up
    Master Master Control Algorithm Group Mbr State Transitions
    ==============================================================================
    1 1 LACP L3_L4 1 Y A 0
    L3_L4 4 Y A 0
    ==============================================================================
    Link State: A-Active, D-Disabled, R-Ready, NP-Port not present, L-Loopback
    Load Sharing Algorithm: (L2) Layer 2 address based
    (L3_L4) Layer 3 address and Layer 4 port based
    Number of load sharing trunks: 1

    ************************************************************************************
    ************************************************************************************

    Core2 # sh lacp

    LACP Up : Yes
    LACP Enabled : Yes
    System MAC : 00:04:96:34:b2:e0
    LACP PDUs dropped on non-LACP ports : 19

    Lag Actor Actor Partner Partner Partner Agg
    Sys-Pri Key MAC Sys-Pri Key Count
    --------------------------------------------------------------------------------
    1 0 0x03e9 00:04:96:34:b2:e1 0 0x03e9 2
    ================================================================================

    Core2 # sh lacp lag 1

    Lag Actor Actor Partner Partner Partner Agg Actor
    Sys-Pri Key MAC Sys-Pri Key Count MAC
    --------------------------------------------------------------------------------
    1 0 0x03e9 00:04:96:34:b2:e1 0 0x03e9 2 00:04:96:34:b2:e0

    Port list:

    Member Port Rx Sel Mux Actor Partner
    Port Priority State Logic State Flags Port
    --------------------------------------------------------------------------------
    1 0 Current Selected Collect-Dist A-GSCD-- 1001
    4 0 Current Selected Collect-Dist A-GSCD-- 1012
    ================================================================================
    Actor Flags: A-Activity, T-Timeout, G-Aggregation, S-Synchronization
    C-Collecting, D-Distributing, F-Defaulted, E-Expired

    ************************************************************************************
    ************************************************************************************

    Core2 # sh lacp lag 1 detail

    Lag Actor Actor Partner Partner Partner Agg Actor
    Sys-Pri Key MAC Sys-Pri Key Count MAC
    --------------------------------------------------------------------------------
    1 0 0x03e9 00:04:96:34:b2:e1 0 0x03e9 2 00:04:96:34:b2:e0

    Up : Yes
    Enabled : Yes
    Unack count : 0
    Wait-for-count : 0
    Current timeout : Long
    Activity mode : Active
    Defaulted Action : Delete
    Receive state : Enabled
    Transmit state : Enabled
    Selected count : 2
    Standby count : 0
    LAG Id flag : Yes
    S.pri:0 , S.id:00:04:96:34:b2:e0, K:0x03e9
    T.pri:0 , T.id:00:04:96:34:b2:e1, L:0x03e9

    Port list:

    Member Port Rx Sel Mux Actor Partner
    Port Priority State Logic State Flags Port
    --------------------------------------------------------------------------------
    1 0 Current Selected Collect-Dist A-GSCD-- 1001
    4 0 Current Selected Collect-Dist A-GSCD-- 1012
    ================================================================================
    Actor Flags: A-Activity, T-Timeout, G-Aggregation, S-Synchronization
    C-Collecting, D-Distributing, F-Defaulted, E-Expired

    ************************************************************************************
    ************************************************************************************
    Core1# show ports sharing
    Load Sharing Monitor
    Config Current Agg Ld Share Ld Share Agg Link Link Up
    Master Master Control Algorithm Group Mbr State Transitions
    ==============================================================================
    1 1 LACP L3_L4 1 Y A 0
    L3_L4 12 Y A 0
    ==============================================================================
    Link State: A-Active, D-Disabled, R-Ready, NP-Port not present, L-Loopback
    Load Sharing Algorithm: (L2) Layer 2 address based
    (L3_L4) Layer 3 address and Layer 4 port based
    Number of load sharing trunks: 1

    currently i don't have the log for Core1, I will provide for the Core1 also. Thanks.



  • 15.  RE: Port Load sharing

    Posted 12-02-2014 06:01
    Paul you have on core2 ports 1,4 in lacp and on core1 ports 1 and 12?


  • 16.  RE: Port Load sharing

    Posted 12-02-2014 06:01
    Hi jarek, Yes correct. It make any different please.


  • 17.  RE: Port Load sharing

    Posted 12-02-2014 06:01
    From show sharing I see that Link Up Transitions in both switches are 0.
    Can you show from both:
    sh lacp counters
    sh configuration | inc shari

    I asked about ports, because I had understood that you set on both switches:
    enable sharing 1 grouping 1,4 algorithm address-based L3_L4 lacp



  • 18.  RE: Port Load sharing

    Posted 12-02-2014 06:01
    Thanks Jarek,

    Core 1
    enable sharing 1 grouping 1, 12 algorithm address-based L3_L4 lacp

    Core 2
    sh configuration | inc shari
    enable sharing 1 grouping 1, 4 algorithm address-based L3_L4 lacp

    will provide the command output tomorrow. thank.



  • 19.  RE: Port Load sharing

    Posted 12-02-2014 11:16
    Try these steps:
    If
    1.configure sharing address-based custom

    check if it load shares

    else
    configure sharing address-based custom hash-algorithm [xor | crc-16 | crc-32]
    try CRC-16----If it doesnt help try CRC-32

    One or the other way changing hashing must help.


  • 20.  RE: Port Load sharing

    Posted 12-02-2014 11:16
    Hi PARTHIBAN,

    I will try tomorrow morning. Thanks a lot for the information. Thanks.


  • 21.  RE: Port Load sharing

    Posted 12-02-2014 11:16
    Hi Parthiban,

    I try this command and not supported in XOS 15.3.3.5 model X450a-24x.

    "configure sharing address-based custom"

    Is there any way to change the CRC to customize the hash for X450a-24x? Thanks.

    BRgds,


  • 22.  RE: Port Load sharing

    Posted 12-02-2014 14:49
    Load sharing is not the same as load balancing. EXOS does load sharing (link aggregation), but not load balancing. As Parthiban and Jarek have mentioned, you can adjust the hash algorithm to attempt to better spread the traffic across the links.

    I'd like to try to understand your use case. Is there a problem with the traffic favoring one link over another, or is it just a preference to have it balanced? What type of traffic is between these two switches? Using the L3_L4 algorithm, you could see one link more saturated than the other if the IP address and protocol port is the same for a majority of the traffic. If you use the L2 algorithm, it will be based on the MAC address of the source and destination systems - if they are the same, only one link will be chosen.


  • 23.  RE: Port Load sharing

    Posted 12-02-2014 14:49
    Hi Drew,

    I have been asking this question in several places. Because this is high priority for me I cannot find the guideline and documentation about this. And I didn't get the appropriate answer to reply my customer. Thank you for your answer.

    Yes we have a big problem with the traffic favoring on only one link.
    Previously, only 1 link saturation experienced.

    We want to load balance the traffic on both link not on a single link. Because users are experienced of network slow and video cannot stream.

    Multicast Traffic for (CCTV video network) is between two core switches.

    I am using the L3_L4 algorithm LACP , But I only see one link more saturated than the other. Link1 is 82 % and Link2 is only 0.25 %

    We want to achieve load balanced on link1 and link2.

    We need your support.

    Please suggest.

    BRgds,
    Paul



  • 24.  RE: Port Load sharing

    Posted 12-02-2014 14:49
    Hi Paul,
    If this is an urgent or high-priority issue, please consider opening a support case with TAC. They will be your best resource for understanding the situation and helping to find a solution in a timely manner.

    If you haven't tried the L2 algorithm, I would recommend making that adjustment. My apologies if you've already mentioned trying that.

    Please keep us updated!

    -Drew


  • 25.  RE: Port Load sharing

    Posted 12-02-2014 14:49
    Thanks Drew,

    Aldy open TAC with high-priority yesterday. still not get the solution and recommendation. So finding the solutions to work around. apology for my quiz are messy in the HUB.

    BRgds,
    Paul


  • 26.  RE: Port Load sharing

    Posted 12-02-2014 14:49
    Hi Paul,
    I found your case number in our system and will let the case owner know about this thread and the urgency.

    We're here to help, no worries on being "messy" :)

    You could try to add a 3rd link to the port group - depending on how many streams there are, the additional link could make a difference in the way they are hashed. If you could upgrade the links to 10G (via XGM module for X450a), then this problem would likely clear itself. Stacking the two switches could also be an option, but would be considerably more difficult to implement since the network is already configured.

    -Drew


  • 27.  RE: Port Load sharing

    Posted 12-02-2014 14:49
    Hi Drew,

    Could you please explain more about upgrade the links to 10G via XGM module for X450a?

    current model is 24-port 1000BASE-X SFP plus 4-port 10/100/1000BASE-T.
    Anyway can we upgrade to 10G uplink?

    BRdgs,
    Paul


  • 28.  RE: Port Load sharing

    Posted 12-02-2014 14:49
    Hi Paul,
    You'll need to purchase a module and optics for installing into the slot on the back of each switch. For the X450a, I would get the XGM2-2sf. It looks like this:


    Once installed, it's a simple configuration to move over to the new 10G link(s).


  • 29.  RE: Port Load sharing

    Posted 12-02-2014 14:49
    Thanks Drew,

    It's very informative and a good solution for users.
    Is it 2x 10G port ?
    Since we are now in this stage.
    Please kindly share the step on simple configuration.
    We need the down time for this module to installation right.


  • 30.  RE: Port Load sharing

    Posted 12-02-2014 14:49
    Hi Paul,
    Yes - that module is 2x 10G SFP+ ports. You will need to power off each switch to install it.

    The configuration is easy to do, but will require looking at the existing VLAN config for the current LAG ports. I feel like that is outside the scope of the community here and would be best left to GTAC to help. If you're already planning on purchasing the modules and optics (be sure to get Extreme Networks Certified Optics!), go ahead and open a new case with GTAC to request help moving the configuration from the two LAG ports to the two new ones - that's the best way to prepare.

    -Drew


  • 31.  RE: Port Load sharing

    Posted 12-02-2014 15:03
    To add to Drews comment: LACPs main use case is to add/improve redundancy. That it does some kind of load sharing is a nice benefit, not more. If real load balancing is necessary some kind of L3 routing protocol, for example OSPF, would be a solution. Some routing protocols (if not all; I am no routing expert) offer "equal path load balancing" which actually looks at the link usage and actively distributes the load.


  • 32.  RE: Port Load sharing

    Posted 12-02-2014 15:32
    As far as I know .Multicast traffic will not be load shared in a lag .
    It will always take one link in a lag.There were known PD's
    may be this feature should have been implemented in new exos release but not till 15.3 I guess


  • 33.  RE: Port Load sharing

    Posted 12-02-2014 15:32
    This statement is misleading. Multicast can be hashed and load shared in a lag, but because much of the traffic is the same source and destination, a single stream can "stick" to one link. Different streams should hash to different links, unless some aspect of the traffic puts them together due to hashing.


  • 34.  RE: Port Load sharing

    Posted 12-02-2014 15:32
    I am 100% sure there was a PD which said multicast traffic will not load share in a lag.

    In 15.6 Concepts guide it is documented that this is supported

    Link Aggregation AlgorithmsSummitStack supports address-based load sharing. (This platform does not support port-based load
    sharing.)
    The following are the types of traffic to which addressed-based algorithms apply and the traffic
    components used to select egress links:
    • Layer 2 frames and non-IP traffic—The source and destination MAC addresses.
    • IPv4 and IPv6 packets
    • L2 algorithm—Layer 2 source and destination MAC addresses. Available on SummitStack and all
    Summit family switches.
    • Broadcast, multicast, and unknown unicast packets (not configurable)—Depends on traffic type:
    • IPv4 and IPv6 packets—The source and destination IP addresses.
    • Non-IP traffic—The source and destination MAC addresses.
    You can control the field examined by the switch for address-based load sharing when the load-sharing
    group is created by using the following command:



  • 35.  RE: Port Load sharing

    Posted 12-02-2014 15:32
    I can't find it at the moment, but that doesn't mean it doesn't exist...

    Since this is an X450a, the last supported version is 15.3.x. Paul, I would suggest first making sure you're on the most recent software for your cores. At the moment, that is EXOS v15.3.4.6-patch1-8. If there's a software bug, it will hopefully be resolved by updating. TAC will need to manage the case beyond that.


  • 36.  RE: Port Load sharing

    Posted 12-02-2014 15:32
    Hi Drew,

    We are not able to purchase a module XGM2-2sf for the X450a. They said it's EOL. Any idea for this where to order please.

    Thanks.


  • 37.  RE: Port Load sharing

    Posted 12-02-2014 15:32
    Hi Paul,
    There are other outside outlets for purchasing older gear that you can find by doing some Googling. You should be able to find an option in your region that carries this part and the optics. If not, your local Extreme Networks sales team may have some other recommendations for upgrade. Sorry I can't be of further help on this question right now.


  • 38.  RE: Port Load sharing

    Posted 03-18-2015 10:33
    Hello guys,

    i can share some experience regarding this case. I have made a sharing group with 6x10G ports between a stack topology with 4 nodes and a X670 switch. The problem was that the first and the second ports were going at peak time up to flat 10G and the other ports at that time were loaded at 5G for example. My XOS version is 15.3.2.11.
    In my case the solution was :
    1. change the custom address-based algorithm from XOR to CRC-16
    2. disable the sharing group and creating new one with address-algorithm custom

    Output from the device:

    configure sharing address-based custom hash-algorithm crc-16
    disable sharing 30
    enable sharing 30 grouping 30-35 algorithm address-based custom lacp

    Hope this was useful !

    Best regards,

    Venko Velev



  • 39.  RE: Port Load sharing

    Posted 04-16-2015 12:55
    @ Venko Velev

    So I assume that after you did the mentioned changes you were able to literally do a load-balance on the links in the LAG group.

    Regards,
    Manish



  • 40.  RE: Port Load sharing

    Posted 08-04-2015 19:40
    Hi Guys,
    This whole thread is quite informative and helped me to correct my understanding on Load sharing on Extreme x450 switches.

    But i have couple of questions.

    Brief:
    In my case, two x450a connected to each other; a static load-sharing (2 physical ports, address based L2) is running and only one of the port is 80% utilized while the 2nd port in LAG has less than 5%. I can understand (or better word is assume) that majority of the traffic has same source and destination MAC, though I do not have wireshark or other logs to support this statement.

    Problematic scenario:
    There were massive outages reported and during troubleshooting on these Extreme switches, it was found that utilization of one port in load sharing touched 100% but for the 2nd port, it remained same as earlier.

    Few of activities performed then:
    1) disable/enable of 2nd port on both switches but no change.
    2) Disable/enable sharing on one switch
    2) added 3rd link in LAG which started carrying the traffic but the older 2nd port of LAG still exhibited same behavior.
    -------
    We then received an update that the issue is resolved. I am now not sure if 2nd or 3rd activity or some activity at any of the other nodes (Not Extreme switches but Application servers- yes there were few activities carried out on Application servers by other teams during the same time) solved the issue.
    ------

    3) We then changed the configuration of LAG from L2 to L3_L4. Traffic pattern and nodes are under observation now.

    Now comes the real question for two configurations ((sharing L2 as well L3_L4).

    1) What will the behavior in case utilization of one of the ports in Load sharing touch 100%?
    2) Will the extra traffic be dropped or shifted to the 2nd port?
    3) How to check if the over utilized port in LAG is dropping traffic?

    Your inputs are much awaited on this case.

    Thanks
    Harkanwal


  • 41.  RE: Port Load sharing

    Posted 08-04-2015 21:04
    Traffic would be dropped. Changing the hashing algo to l3_l4 is a good try. If most of your traffic is between 2 end-systems, you need to find some entropy in the headers to help load-balancing. L4 usually helps there.


  • 42.  RE: Port Load sharing

    Posted 08-04-2015 22:37
    In addition to comments from Stephane, the packets at the port level can be monitored using the command, "show port congestion".


  • 43.  RE: Port Load sharing

    Posted 09-01-2015 04:53
    Router---------Extreme switch(X450a-48t)--------(Port aggregation)-------Extreme Switch (Summit48si),

    On X450a-48t (enable sharing 2:2 grouping 2:2-4 algorithm address-based L3_L4) and on Summit48si (enable sharing 1 grouping 1,2,3 algorithm address-based )

    We are facing congestion on all aggregated ports (X450a-48t) while not able to see on Summit48si. There is sufficient bandwidth. What could be the problem



  • 44.  RE: Port Load sharing

    Posted 09-01-2015 07:18
    Hi,

    congestion counter increase when switch buffer on egress have insufficient space for packets.

    Mostly traffic flows from router to access switch and because of that you see congestion only on X450a.
    For example you have about 500 Mb egress traffic per 1G port.
    Internet traffic is not constant, some time we see micoroburst - we try send more than 1G or near 1G

    What you see after: 'show port buffer' ?

    What you can do:
    - enable flow control
    - if you use qosprofiles - delete unused qosprofiles
    - tune port buffer - don't change if you don't know what are you doing :)

    --
    Jarek