Redundant connection from Service provider causes loops

[Keith9]

Our service provider dropped a 10 gig private ring into our rack at our colocation facility.  At our request, we asked for two ports so we can take one to each core switch (x690's).  Unfortunately the handoff device that the service provider racked, (ADVA FSP 150-XG304) doesn't support LACP.  They call this a mirrored uplink port.

If both ports are enabled, it obviously loops the network.  Any other switch on this service provider without DOS protection falls off the network.  Needless to say we have one port disabled for now, and remediated one location that didn't have enable dos protection set.

So we tried the ELRP and for some reason it blocks port 65, the ISC between both core switches instead of port 48 of one of the uplinks.  RSTP, MSTP same results.  Doesn't seem to fully protect the loop.  So the next thought is what if we take this transport VLAN OFF of the ISC link between both switches?  Then each core switch is an independant brain.  They both have their own IP Address and can just peer back through the providers fully transparent (to us) switch via their network to our other locations.  I know for servers that have MLAG connections, they won't be connected to this vlan directly anyway.  This is just a handoff vlan we have common on all our switch uplink ports to the service provider.  They all OSPF peer with BFD enabled between the the private fiber connection vs a higher cost IPSEC VPN tunnel over the internet with an adjacency established via a firewall running OSPF.

I've suggested to the service provider maybe going with a Ciena 3924, which specifically shows on their datasheet LACP supported on the customer side, and continued 10g fiber ring diversity on the service provider site.  Our project coordinator with the serivce provider thanked me and will take that up the chain, but for us, every site has this ADVA switch and I don't think they'll change it.  Maybe they will explore it for future customers but right now we can only take the equipment they will provide.

Attaching is a diagram.  Parts in red are new.  Everything else is exsiting and has been working for quite some time.

What makes this site different is trying to utilize two "dumb" transparent 10g links from the ISP's switch (Ring D in this picture) to two independant x690 switches.  The firewall, servers, storage, and anything that can be.... are connected via defined MLAGs to these two switches and in their apporopriate vlans.  Only vlan 102 needs to be on switches that participate in peering on this network.

[Keith9 · Accepted solution]

This is what we ended up doing.

We took the transport vlan (102) OFF the switch to switch port.
So from our metro-e ring vlan 102 and all single switch uplinks to the service provider... DC Switch 1 and DC Switch 2 both join it individually, and since its not tagged on the switch to switch port 65, it doesn't loop.  They peer in OSPF independantly (192.168.102.11 and 192.168.102.12).  Locally VRRP is the gateways out between vlans that exist acrros the two switches, but OSPF costing determines which 10g switch path it will take out.

We did some testing and when pulling a link to one switch, theres little to no fanfare.  Maybe 1 ping lost, but BFD keeps things active very quick.

[Stefan_K_]

They don't support LACP but static LAG, so that can be used to form the MLAG.

[Keith9]

Oh and here is logs from one of the switches IF we would enable both links from Ring D to DC-SW1 and DC-SW2.
Ring B (192.168.102.30) didn't have DOS protection so it fell of the grid for a minute while we tested, but we enabled it now!

03/27/2024 14:48:31.14 <Noti:ospf.neighbor.ChgState> Changing the state of neighbor rtid 192.168.255.2 ipa 192.168.102.1 to state = FULL due to Loading done.
03/27/2024 14:48:31.14 <Noti:ospf.neighbor.ChgState> Changing the state of neighbor rtid 192.168.255.2 ipa 192.168.102.1 to state = LOADING due to exchange done event.
03/27/2024 14:48:31.13 <Noti:ospf.neighbor.ChgState> Changing the state of neighbor rtid 192.168.255.2 ipa 192.168.102.1 to state = EXCHANGE due to negotiation done event.
03/27/2024 14:48:31.13 <Noti:ospf.neighbor.ChgState> Changing the state of neighbor rtid 192.168.255.2 ipa 192.168.102.1 to state = EX_START due to AdjOK event.
03/27/2024 14:48:30.13 <Noti:ospf.neighbor.ChgState> Changing the state of neighbor rtid 192.168.255.2 ipa 192.168.102.1 to state = 2WAY due to two way event.
03/27/2024 14:48:26.13 <Noti:ospf.neighbor.ChgState> Changing the state of neighbor rtid 192.168.255.100 ipa 192.168.102.100 to state = 2WAY due to adjacency getting destroyed.
03/27/2024 14:48:25.13 <Noti:ospf.neighbor.ChgState> Changing the state of neighbor rtid 192.168.255.100 ipa 192.168.102.100 to state = EX_START due to AdjOK event.
03/27/2024 14:48:25.12 <Noti:ospf.neighbor.ChgState> Changing the state of neighbor rtid 192.168.255.2 ipa 192.168.102.1 to state = INIT due to one way event.
03/27/2024 14:48:24.88 <Info:DOSProt.DelACLOK> Removed ACL from port 1:57, srcIP 0.0.0.0 to destIP 224.0.0.5, protocol any
03/27/2024 14:48:24.66 <Noti:ospf.neighbor.ChgState> Changing the state of neighbor rtid 192.168.255.30 ipa 192.168.102.30 to state = FULL due to Loading done.
03/27/2024 14:48:24.66 <Noti:ospf.neighbor.ChgState> Changing the state of neighbor rtid 192.168.255.30 ipa 192.168.102.30 to state = LOADING due to exchange done event.
03/27/2024 14:48:24.64 <Noti:ospf.neighbor.ChgState> Changing the state of neighbor rtid 192.168.255.30 ipa 192.168.102.30 to state = EXCHANGE due to negotiation done event.
03/27/2024 14:48:24.63 <Noti:ospf.neighbor.ChgState> Changing the state of neighbor rtid 192.168.255.30 ipa 192.168.102.30 to state = EX_START due to bad LS request.
03/27/2024 14:48:24.63 <Warn:ospf.neighbor.ProcDDFail> Processing DD from neighbor 192.168.255.30 fails on NBR seqmismatch event,case else.
03/27/2024 14:48:21.48 <Info:DOSProt.PktCntExcd> Notify-threshold for L3 Protect packet count of 3500 reached
03/27/2024 14:48:18.78 <Info:DOSProt.AddACLOK> Added an ACL to port 1:57, srcIP 0.0.0.0 to destIP 224.0.0.5, protocol any
03/27/2024 14:48:18.71 <Info:DOSProt.PktCntExcd> Notify-threshold for L3 Protect packet count of 3500 reached
03/27/2024 14:45:08.81 <Noti:ospf.neighbor.ChgState> Changing the state of neighbor rtid 192.168.255.11 ipa 192.168.102.11 to state = 2WAY due to two way event.
03/27/2024 14:45:08.81 <Noti:ospf.neighbor.ChgState> Changing the state of neighbor rtid 192.168.255.11 ipa 192.168.102.11 to state = INIT due to hello received.
03/27/2024 14:45:08.81 <Noti:ospf.neighbor.ChgState> Changing the state of neighbor rtid 192.168.255.11 ipa 0.0.0.0 to state = DOWN due to new neighbor.
03/27/2024 14:44:52.41 <Noti:ospf.neighbor.ChgState> Changing the state of neighbor rtid 192.168.255.11 ipa 192.168.102.11 to state = DOWN due to inactivity timer expiry.
03/27/2024 14:44:11.41 <Noti:ospf.neighbor.ChgState> Changing the state of neighbor rtid 192.168.255.11 ipa 192.168.102.11 to state = 2WAY due to two way event.
03/27/2024 14:44:09.35 <Noti:ospf.neighbor.ChgState> Changing the state of neighbor rtid 192.168.255.11 ipa 192.168.102.11 to state = INIT due to one way event.
03/27/2024 14:42:43.20 <Noti:ospf.neighbor.ChgState> Changing the state of neighbor rtid 192.168.255.11 ipa 192.168.102.11 to state = 2WAY due to two way event.
03/27/2024 14:42:43.20 <Noti:ospf.neighbor.ChgState> Changing the state of neighbor rtid 192.168.255.11 ipa 192.168.102.11 to state = INIT due to hello received.
03/27/2024 14:42:43.20 <Noti:ospf.neighbor.ChgState> Changing the state of neighbor rtid 192.168.255.11 ipa 0.0.0.0 to state = DOWN due to new neighbor.
03/27/2024 14:42:16.41 <Noti:ospf.neighbor.ChgState> Changing the state of neighbor rtid 192.168.255.11 ipa 192.168.102.11 to state = DOWN due to inactivity timer expiry.
03/27/2024 14:15:42.18 <Info:DOSProt.DelACLOK> Removed ACL from port 1:57, srcIP 0.0.0.0 to destIP 224.0.0.5, protocol any
03/27/2024 14:15:35.87 <Info:DOSProt.AddACLOK> Added an ACL to port 1:57, srcIP 0.0.0.0 to destIP 224.0.0.5, protocol any
03/27/2024 14:15:35.81 <Info:DOSProt.PktCntExcd> Notify-threshold for L3 Protect packet count of 3500 reached
03/27/2024 14:13:32.13 <Info:DOSProt.DelACLOK> Removed ACL from port 1:57, srcIP 0.0.0.0 to destIP 224.0.0.5, protocol any
03/27/2024 14:13:25.58 <Info:DOSProt.AddACLOK> Added an ACL to port 1:57, srcIP 0.0.0.0 to destIP 224.0.0.5, protocol any
03/27/2024 14:13:25.53 <Info:DOSProt.PktCntExcd> Notify-threshold for L3 Protect packet count of 3500 reached
03/27/2024 14:03:32.16 <Info:DOSProt.DelACLOK> Removed ACL from port 1:57, srcIP 0.0.0.0 to destIP 224.0.0.5, protocol any
03/27/2024 14:03:31.16 <Info:DOSProt.DelACLOK> Removed ACL from port 1:57, srcIP 192.168.102.100 to destIP 224.0.0.5, protocol any
03/27/2024 14:03:25.90 <Info:DOSProt.AddACLOK> Added an ACL to port 1:57, srcIP 0.0.0.0 to destIP 224.0.0.5, protocol any
03/27/2024 14:03:25.83 <Info:DOSProt.PktCntExcd> Notify-threshold for L3 Protect packet count of 3500 reached
03/27/2024 14:03:25.05 <Info:DOSProt.AddACLOK> Added an ACL to port 1:57, srcIP 192.168.102.100 to destIP 224.0.0.5, protocol any
03/27/2024 14:03:24.95 <Info:DOSProt.PktCntExcd> Notify-threshold for L3 Protect packet count of 3500 reached
03/27/2024 13:51:02.16 <Info:DOSProt.DelACLOK> Removed ACL from port 1:57, srcIP 0.0.0.0 to destIP 224.0.0.5, protocol any
03/27/2024 13:51:01.16 <Info:DOSProt.DelACLOK> Removed ACL from port 1:57, srcIP 192.168.102.100 to destIP 224.0.0.5, protocol any
03/27/2024 13:50:56.55 <Info:DOSProt.PtrnNotFnd> No traffic pattern found
03/27/2024 13:50:56.44 <Info:DOSProt.PktCntExcd> Notify-threshold for L3 Protect packet count of 3500 reached
03/27/2024 13:50:55.38 <Info:DOSProt.AddACLOK> Added an ACL to port 1:57, srcIP 0.0.0.0 to destIP 224.0.0.5, protocol any
03/27/2024 13:50:55.30 <Info:DOSProt.PktCntExcd> Notify-threshold for L3 Protect packet count of 3500 reached
03/27/2024 13:50:54.48 <Info:DOSProt.AddACLOK> Added an ACL to port 1:57, srcIP 192.168.102.100 to destIP 224.0.0.5, protocol any
03/27/2024 13:50:54.38 <Info:DOSProt.PktCntExcd> Notify-threshold for L3 Protect packet count of 3500 reached
03/27/2024 13:31:28.45 <Info:DOSProt.DelACLOK> Removed ACL from port 1:57, srcIP 0.0.0.0 to destIP 224.0.0.5, protocol any
03/27/2024 13:31:25.45 <Info:DOSProt.DelACLOK> Removed ACL from port 1:57, srcIP 192.168.102.1 to destIP 224.0.0.5, protocol any
03/27/2024 13:31:21.89 <Info:DOSProt.AddACLOK> Added an ACL to port 1:57, srcIP 0.0.0.0 to destIP 224.0.0.5, protocol any
03/27/2024 13:31:21.83 <Info:DOSProt.PktCntExcd> Notify-threshold for L3 Protect packet count of 3500 reached
03/27/2024 13:31:18.95 <Info:DOSProt.AddACLOK> Added an ACL to port 1:57, srcIP 192.168.102.1 to destIP 224.0.0.5, protocol any
03/27/2024 13:31:18.83 <Info:DOSProt.PktCntExcd> Notify-threshold for L3 Protect packet count of 3500 reached