topic RE: Fabric Attach pros/cons in ExtremeSwitching (VSP/Fabric Engine)

Fabric Attach pros/cons

Anonymous — Fri, 17 Dec 2021 08:34:00 GMT

Hi all,

I'm halfway through my first Fabric Connect deployment, and looking for in the field experience with Fabric Attach.
At this network's core are two server rooms, with a VSP 7400 and a 5520-24t running VOSS in each one, then we have X450G2 stacks for distribution, along with a few old B5 stacks.
Software versions are VOSS 8.4.2.0 and EXOS 30.7.2.1.

So, Fabric Attach.
Automatically pulling VLANs from the edge switches sounded great initially.

That is, until I realized that:
- FA is dog slow, VLANs get mapped about 1 min after you link comes up, with default LLDP timings
- FA requires full trust in your edge switches, as any I-SID/VLAN in your core is "attachable", and apparently can't be protected from it
- FA cannot handle multiple paths, but relies on LACP for redundancy, so I still need to set this up on both core and edge
- and I still have to deal with port-based VLANs on VOSS anyway, for B5 stacks, servers, routers, etc

Lowering the LLDP transmit interval from 30s to 10s only made FA barely usable, mapping VLANs still takes from 15s to 20s.
I consider using a smaller LLDP interval to be unreasonable, as it's a global value for all ports in the EXOS stack, I and don't want this to affect other devices in any way.
Didn't find any KB article dealing with this issue.
I guess the problem is LLDP advertisements are sent in sync globally to all ports, and we end-up waiting for the next scheduled dispatch, and that's what makes it so painfully slow, but really FA on the edge should trigger a "gratuitous" LLDP advertisement when the link comes up and be done with it.
So long for the Fabric Connect super fast convergence time...

Then, the lack of any core-side control over which VLAN/I-SID can be mapped really annoys me.
We do use FA authentication (no one should use unauth'ed FA in its current state IMO), but that's not enough.
I don't want the edge switches to be able to extend the server or infrastructure VLANs, it just doesn't feel safe.
I can see the value in only provisioning edge VLANs in only one place, but I want to be able to selectively enable VLANs/I-SIDs for FA on the core/server side.

And ultimately, I don't feel that I'm really gaining that much from FA in return.
I still have to configure core to edge ports for MLT/LACP (maybe even switch to trunk mode manually, I don't remember).
I still need to map I-SIDs to actual VLANs on the 7400s in order to provide IP subnets gateways, routing and so on (so I end-up with a double I-SID to VLAN mapping).

Did I miss something here?
Does FA bring any other benefit?
Is there a well hidden trick to make VLAN mappings happen faster?
Is there a way to whitelist I-SIDs for FA? (and prevent any other I-SID from being ever mapped)
Does FA really only make sense in a "real" larger Fabric Connect network?
Am I wrong in considering dropping it in favor of static VLANs?

Thanks,

Nicolas

RE: Fabric Attach pros/cons

Miguel-Angel_RO — Sat, 18 Dec 2021 18:55:00 GMT

Hi Nicolas,

A lot of questions there.
Let's answer the main one: "Am I wrong in considering dropping it in favor of static VLANs?"
It depends of you business requirements. For some environments, the FA process is indeed seen as too slow while on others it is ok.
By experience, the standard timers are ok for most of the customers.

Here few comments:
I would urgently upgrade to 8.4.2.1, the 8.4.2.0 has an ugly bug and has been removed from the support site.

On the last EXOS releases you also have the "Fabric Attach Automatic LAG Creation". This automates the LAG creation and VLANs mapping between EXOS and SMLT clusters.
I remember one hospital who cannot cope with the base timers and went for static LAG configuration.

You can specify a white list of i-sid's allowed via radius authentication, see here : https://extremeportal.force.com/ExtrArticleDetail?an=000073262

On the edge switches you can have a minimalistic config with netlogin on all ports.
The radius will then return the couple VLAN/I-SID (or a role) on an authentication based approach for the clients.

With those features you have a full automated port configuration.
This is not solving the lag time to have the VLAN provisioning but this is happening only for the first client. The second client on the same VLAN will just get the VLAN on the port activated, the uplink will be already provisioned.

I hope this helps,
Mig

RE: Fabric Attach pros/cons

EXTR_Paul — Sun, 19 Dec 2021 19:43:00 GMT

I would consider upgrading to your EXOS switch to 31.5 and see if you can replicate the issue.
Also are you VSPs core devices in vIST clusters?

to answer some of your questions.
- FA is dog slow.
FA is not a resiliency protocol. Its an automation/ztp feature. FA needs to go through a ton of order of operations to make sure that new switch get adopted proper. If you want to speed it up, make sure Spanning Tree is disabled on your VSP ports.   FA won't start its Mojo until spanning tree is done its business. I usually see the FA assignments kick in after 10-20 seconds.

- FA requires full trust in your edge switches, as any I-SID/VLAN in your core is "attachable", and apparently can't be protected from it.

FA gives the user granularity in the moves/add/changes in vlans in the edge. The VLAN needs to exist on the edge switch and the VLAN needs an explicit i-sid. This can be done manually or with XIQ-SE/Control. This is full control.    If you think about how some vendors do tagging. Some automatically put all VLANs on all trunks and one needs to explicitly prune the vlans away. Or protocols like VTP that just send VLANs everywhere.

FA also has security with authentication so only trusted edge switches can receive the FA assignments and elements

- FA cannot handle multiple paths, but relies on LACP for redundancy, so I still need to set this up on both core and edge.

Not true. You can manually create a LAG group on the EXOS switch.   Also, with EXOS 31.X there is an auto-lag feature that will sense that the two or more links are connected to an MLT/SMLT from the VSP and will automatically create the lag group on the EXOS switch. I have tested this a lot. It actually works very fast. But you need to be running the newest code.

- and I still have to deal with port-based VLANs on VOSS anyway, for B5 stacks, servers, routers, etc.

Not necessarily. FA will auto create VLANs in the core.
But usually network admins want that control.

RE: Fabric Attach pros/cons

Roger_Lapuh — Mon, 20 Dec 2021 10:58:00 GMT

Nicolas
we are aware that in some scenarios we are relying on LLDP timers to signal service bindings. We will be addressing that by sending triggered updates (LLDP). Stay tuned, we hope to have this addressed by 1H 2022. There will be SW updates required on VOSS and EXOS.

Roger

RE: Fabric Attach pros/cons

Anonymous — Tue, 21 Dec 2021 03:46:00 GMT

Hello Mig,

Yes, I guess for many/most customers, VLAN automation is worth the delay.
My use case is a medical devices factory, where a longer than expected interruption will cause a chain reaction and escalate very quickly.

Thanks for the reminder about the 8.4.2.0 DHCP issue, upgrade is planned already for the next maintenance window, i.e. next week.

OK, I missed the automatic LAG creation in EXOS 31.x.
I think I'll still stay on 30.7 for now (actually upgrading from 22.7 as part of the network upgrade) until 31.x gets more widely spread.

I'm only using FA between X450G2 stacks and VSP7400's, not sure if provisionning I-SIDs over RADIUS would work here, but I don't feel like making the network infrastructure (vs. user devices) rely on a RADIUS server.
Already using ExtremeControl NAC on X450G2 with netlogin on all ports, but you I think you have more of an ERS scenario in your mind, don't you?

RE: Fabric Attach pros/cons

Anonymous — Tue, 21 Dec 2021 03:48:00 GMT

Hello Paul,

No, I did not try running 31.x yet, and I'll stay on 30.7 for now, but will definitely give a try as time permits.

> If you want to speed it up, make sure Spanning Tree is disabled on your VSP ports.
But I do want STP! 🙂 I actually got hit hard last week by the lack of STP support for "multihomed" devices in the default VSP configuration (https://extremeportal.force.com/ExtrArticleDetail?an=000082836)
The 30-60 seconds delay is clearly not due to STP though, ports are up in forwarding state loooong before VLANs get mapped by FA.
Also have a deep hate for Cisco's "help me I'm stuck in the 90s" VLANs, but TBH, VOSS makes it difficult and painful as well, especially compared to EOS/EXOS.

Yes, I get your arguments for FA.
I still believe you should not be allowed to pull just any VLAN in your core network from an edge device.

Thanks for telling about the automatic LAG creation in EXOS 31.7 as Mig did, and glad it worked for you.
I need to look into this.

> > and I still have to deal with port-based VLANs on VOSS anyway, for B5 stacks, servers, routers, etc.
> Not necessarily. FA will auto create VLANs in the core.
Unless I missed something, FA won't help with non FA-enabled devices.

RE: Fabric Attach pros/cons

Anonymous — Tue, 21 Dec 2021 03:51:00 GMT

Roger,

Thanks for the good news!
I'm really looking forward to this triggered LLDP updates feature.
A global "fa allowed-vlans" command (only allow specified VLANs to be pulled by FA) is also high on my Xmas wish list. 😄

RE: Fabric Attach pros/cons

Anonymous — Tue, 21 Dec 2021 20:18:00 GMT

More testing with the FA/LLDP VLAN provionning delay on EXOS.
I did not see any noticeable difference in FA provisionning time between EXOS 22.7.3.5, 30.7.2.1 and 31.5.1.6.
It was not as bad as I said before: about 36-37 seconds, with the default LLDP transmit interval, in my environment (I may have hit another issue when I consistenly got a 1 min delay initially).
But that's still way too slow if you ask me.

I also tried explicitly configuring the management I-SID/VLAN on the port/MLT in VOSS (fa management i-sid <isid> c-vid <vlanid>).
I guess this feature is really aiming at provisionning native FA devices (cams, APs, ..), and I don't have any, but it was still worth a try.
(Also EXOS has a similar command, "configure fabric attach management vlan", which seems to only makes sense in standalone proxy mode.)
As could be expected, this configuration does make the said management VLAN come up almost instantly, as VOSS will map it without waiting for LLDP advertisements to be sent back and forth, but this does not affect the provisionning delay for the other VLANs.

So, I'll be sticking with EXOS 30.7 with static VLANs for now.
Thanks all for your help.

RE: Fabric Attach pros/cons

Evert — Fri, 19 Aug 2022 15:59:00 GMT

Hi Roger,

Any news on the triggered LLDP updates? I couldn't find mention of this in release notes of VOSS 8.6 or 8.7.

We ran into an issue with devices for which the FA takes too long, the device has given up trying to reach its gateway before traffic is passed on the uplink of the exos switch (the FA vlan assingment is pending).

Thank you in advance!
Evert.

RE: Fabric Attach pros/cons

Roger_Lapuh — Fri, 19 Aug 2022 16:54:00 GMT

Hi Evert
yes Release 8.8 will support this from the VSP perspective. GA end of August. Roger

RE: Fabric Attach pros/cons

Evert — Fri, 19 Aug 2022 18:48:00 GMT

Hello Roger,
That's good news, thank you!