topic Re: EXOS MLAG/LACP question / clarification in ExtremeSwitching (EXOS/Switch Engine)

EXOS MLAG/LACP question / clarification

Frank — Fri, 17 May 2019 16:51:34 GMT

I'm trying to get a clearer understanding of when to use MLAG with LACP and when not. I did read the note in https://extremeportal.force.com/ExtrArticleDetail?an=000079895 that reads "if the downstream device (e.g. switch, server, etc..) is using LACP LAG sharing on its ports, it will be necessary to enable LACP LAG on the MLAG ports", but I'm still a little fuzzy - probably because I don't understand the "black magic" fully.

Here's my layout - we're running EXOS 16.x:
ISC (ports 1:24/2:24) BD8800-1 ==== BD8800-2 1:1| 2:1\ 1:1/ |2:1 | \ / | | X | | / \ | 23| 24/ 23\ |24 Switch-1 Switch-2 5| 5| | | | | Server (Linux or Microsoft) with LACP share

Shared port group between the BD8800s for the ISC (enable sharing 1:24 grouping 1:24,2:24)
1-port shared "port group" to Switch-1 on the 8800 (enable sharing 1:1 grouping 1:1)
Same to Switch 2 (enable sharing 2:1 grouping 2:1)
Shared port group on Switch-1/2 to the BD8800s (sharing 23 grouping 23,24)
Single-port share-group on Switch-1 and Switch-2 on the port 5 to the server

mlag: enable mlag port 1:1 peer "8800-1/2" id 123 (on the 8800s)

On top of that: The BD8800s are the "default gateway" in a VRRP configuration for the vlan that the attached server is in (in case it matters)

Question 1:
In the scenario above, what shares on which switches/shares would have to be done with LACP - for instance "enable sharing X grouping ... L3_L4 LACP" (since I'd like to use L3_L4 on the server side)?

Question 2:
If I were to add an ISC link between Switch-1 and Switch-2, would the only LACP share configuration be on Switch-1/2 (and not on BDs)?

Question 3:
Are there any advantages/disadvantages to using "LACP" as opposed to not using it and going with Extreme's "pure purple magic"?

Thank you for looking at this!

Re: EXOS MLAG/LACP question / clarification

OscarK — Fri, 17 May 2019 18:53:20 GMT

The only thing LACP does is negotiate members of a LAG.
LACP will make sure the right ports are in the LAG and patch mistakes are not causing weird problems as the ports will not join the LAG.

LACP does not do anything more, it does not determine any automatic load sharing alghorithm, that is configured manually with the configured sharing algorithm.

The only disadvantage of LACP is that the lacp packets are processed by the CPU. If the CPU gets really busy or congested LACP packets might get lost resulting in ports being removed from the LAG.

Re: EXOS MLAG/LACP question / clarification

Frank — Fri, 17 May 2019 20:08:48 GMT

Oh, so all I'd need to do is to configure on Switch-1 and 2 something like

code:

enable sharing 5 grouping 5 l2_l3 lacp

And then I'm done with it? That would be downright easy!

Thank you so much for the easy to understand explanations - I got stuck in a loop between "overthinking", "read all kinds of technical docs" and "head exploded".

Re: EXOS MLAG/LACP question / clarification

David_Choi — Sun, 19 May 2019 09:49:16 GMT

Hi Frank,

It looks like the switch-1/2 are not configured with MLAG currently. Is it correct? As they are not in MLAG, you can not configure LACP on server side. This is because the switch-1 and 2 are physically / logically different hardware.

Re: EXOS MLAG/LACP question / clarification

Frank — Mon, 20 May 2019 16:44:48 GMT

David, I'm not sure I fully understand. You're correct in that (currently) Switch-1/2 do not have an ISC link between them. I thought the mlag on the BD8800 would/should take care of the proper packet paths.

Let's say I have a full two-tier MLAG configuration, meaning an ISC link between switch1/2, appropriate config changes on the BD8800s (need to share/group 1:1 and 1:2), and of course define an mlag port 23 peer switch1/2:
What would happen if the ISC link between switch-1/2 dies? Wouldn't I be in the same situation as I'm in now?

Re: EXOS MLAG/LACP question / clarification

Tomasz — Mon, 20 May 2019 22:19:31 GMT

Hi Frank,

If your Switch1 and 2 had an ISC, you can create MLAG, so from the Server perspective they will be a single device (virtual MAC, somewhat similar to virtual IP with VRRP concept). Then there is a point for aggregating uplink from Server to Switch1 and Switch2. At the moment (as per your diagram) this LACP on server side shouldn't be fully sucessful as on both ends the Server will see two different LACP keys (while it needs one, what is achievable thru MLAG).
Your 2-tier MLAG between switches and BDs is quite nice, but it doesn't do anything with Switches/Server communication. If you had ISC between the switches, you'd take advantage and have a possibility to terminate LACP/LAG from your server to two separate switches and have it presented to the Server as single logical link, that's what MLAG is about. 😉
Having situation like right now, LACP won't work 100% and static LAG would work (I'm only concerned about having the same MAC address learnt in two separate spots), but you might also experience sub-optimal flow handling - ISC would also pass data VLANs, so whenever some packet comes from the Server due to hashing algorithm to the left switch, but something is attached to the right switch, it can travel across ISC directly to the right switch without doing some workarounds and increasing vertical links utilization.
Once your ISC breaks, some sort of disruption might be presented AFAIK (e.g. FDB Checkpointing that is about p2p exchange of learned source MAC addresses). That's why it is a good practice to have it built on aggregation as well. And, just as a sneak peek, in SPB-based Fabric networks there is a feature called 'Virtual IST' that allows to run MLAG (called SMLT) successfully as long as there's ANY connection between peers. Cool, huh? 🙂
Another example: once I talked to a customer who had 'officially approved' two-tier MLAG design with only ISCs and vertical links, without diagonal ones. I was struggling with myself to eventually see, that yes - it will work, but like having the MLAG in failed links condition, so less bandwidth and resiliency in fact.
If it's not an issue, just go for an ISC between Switch1 and Switch2.

Hope that helps,
Tomasz

Re: EXOS MLAG/LACP question / clarification

Frank — Tue, 21 May 2019 15:09:43 GMT

Thomasz, thank you for your detailed and insightful answer!

Final (hopefully) question: Since I'm "only" building an ISC/MLAG link between Switch-1 and Switch-2 to accommodate LACP to a device attached to Switch-1 and Switch-2, do I have to go full-blown two-tier MLAG, or can I treat it as a separate mlag, just for "port-5". I admit, I haven't fully thought it through, but if I need to build a full two-tier MLAG, I'll have to have a real service window when I put ports 1:1/1:2 on the BD8800s in a share.
What I mean is: create an ISC, mlag port(s) 5, but don't tag port 5's vlan (data-vlan) to the ISC port. In my case, "port 5" is the only port in that vlan on switch1/2 (except for the 23/24 share going up to the BDs), anything else that would be in that vlan is on other switches that are attached to the BDs

If I can get away with a "this is a separate mlag", then things become simpler from a scheduling aspect.

I have a nagging feeling that the "separate mlag" option, where I just mlag only for the LACP links is going to trip me up in the future when I see "ISC, oh, just handle it like everywhere else and tag all the vlans through it as well", so I realize I really should do full-two-tier at some point "soon".

Re: EXOS MLAG/LACP question / clarification

Frank — Tue, 21 May 2019 18:36:14 GMT

I think the documentation just hit me over the head:

You must add the ISC to every VLAN that has an MLAG link as a member port

For my config, I'll take that to mean "Full two-tier MLAG - or things won't work right"

Re: EXOS MLAG/LACP question / clarification

Tomasz — Wed, 22 May 2019 01:34:52 GMT

Hi Frank,

Just to make sure, do you have the 2-tier MLAG deployed already there, or is it just a concept for more links and higher redundancy at the moment?

Every MLAG instance is a unique thing. You could do only BDs-to-Switches MLAGs, one for each Switch#, with all (or some set of) VLANs traversing north-south and thru ISC. Having two-tier MLAG (the former and another one from Switches to BD perspective) gives you complete MLAG-based redundancy that is possible with two pairs of devices on each tier. I believe it's good to have 2-tier MLAG in place with your scenario.

If I understood your concern properly, your Switch 1 and 2 -based MLAG for the Server can be treated independently from two-tier (and MLAG to any other server would be a separate MLAG instance as well, they can utilize the same physical links for ISC though). You can have 2-tier MLAG on top and not work the same way with your server links, you can have MLAG for servers link redundancy without 2-tier above, or you can have both.
Full MLAG in your diagram would mean to have two MLAG configurations in 2-tier and third one on server access side, 3 MLAG 'deployments' total. Make sure to pass the right VLANs in all of the aggregated links and the ISCs.

If you take a look at EXOS User Guide chapter 6 (https://www.extremenetworks.com/support/documentation/extremexos-software-22-5/) you may find configuration guidelines and an example that is relevant to your diagram as well, please see pages 254-257 (the last page shows your example).

Hope that helps,
Tomasz

Re: EXOS MLAG/LACP question / clarification

Frank — Wed, 22 May 2019 15:33:42 GMT

Thank you all very much!
I think I (finally) got it 🙂
Frank