Hi!
We’re considering using multiple pairs of X590 setup as MLAG peers. Looking at one pair (one MLAG peer pair), what would be the best way to build it considering the lack of 40/100 G ports? The X590 has three 40/100 G ports but all three can only be used if it is stacked and then two are used for stacking, leaving one per switch for uplink. We like using MLAG peers as they are autonomous and can be individually rebooted, which is not the case with a stack.
Options (including, but not limited to...):
1 - Full mesh MLAG to core using the two 40/100 G ports as uplinks and n x 10 G ports for the ISC link between the two X590’s.
2 - Non-full mesh MLAG from the X590’s to the core. X590 switch A will then have its uplink to core switch A and X590 sw B vill have one uplink to core bw B. This allows for a 40/100 G port between the two and we’d then use an alternate ISC (ISC2) to cover for the failure case that the single ISC goes down.
There are pros and cons with both scenarios, but the main drawback with #1 is the asymmetry in the ISC (say 4 x 10 G) compared to the uplinks (40 or 100 G) and for #2 it is that a single uplink failure may cause longer delays than we like.
Servers are OK with longer delays if an uplink goes down, but storage is not. On the other hand the slow links as ISC in #1 may constrain traffic in some cases. I’d like to know if anyone has experience in the traffic flow between two MLAG peers and why there is traffic on those links from time to time.
All uplinks are symmetric and redundant obviously and go to an overlying core (X870’s in an MLAG config in this case). Loads are mainly VMware servers with at least one connection to each X590 in the MLAG pair, using VMware’s standard load sharing (not LAG/LACP) so VMs are either present on one or the other switch. This may cause traffic loads on the ISC link if one server that happens to be routed via the port connected to X590 A wants to talk to another VM on another VMware server and this VM is routed to a port on X590 B. (Routed in terms of that VM being connected to that interface in the VMware internal switch, not IP routed.) All traffic loads going to/from the core will probably never go via the ISC unless we have a failed link. Some servers will be LAG:ed to the X590’s (one link to each X590), but those are probably not our main concern.
So: which one is the best ? Answers like “How long is a string” are not accepted… Perhaps you’d like to present scenario C? What would that be? Please elaborate!
Best regards,
/Fredrik
