topic Re: Fabric Extend over ISP (Hub-and-spoke) in Network Architecture & Design

Fabric Extend over ISP (Hub-and-spoke)

bfaltys — Fri, 21 May 2021 02:45:19 GMT

We have our own fiber mesh and SPBM running fine. We are getting additional links at multiple sites from an ISP. The ISP is using q-in-q so we basically have a pseudowire from the core to each site. Since we cannot map our 2 BVLANs to multiple ISP tags we need to use layer 3 fabric extend, as far as I understand. It seems like this is possible, but I haven’t been able to locate a good example yet. Am I thinking about this wrong? Would this require our interface to the ISP at both ends to be a layer 3, routed interface? Would we need a separate /30 between the core and each site? I’m sure I have more questions, but this would be a good start.

Re: Fabric Extend over ISP (Hub-and-spoke)

bfaltys — Fri, 21 May 2021 02:46:15 GMT

This image just shows one “spoke” site.

Re: Fabric Extend over ISP (Hub-and-spoke)

Miguel-Angel_RO — Sun, 23 May 2021 20:06:47 GMT

Hi bfaltys,

L3 interfaces are not needed.

Here from the doc:

Mig

Re: Fabric Extend over ISP (Hub-and-spoke)

bfaltys — Mon, 24 May 2021 22:35:46 GMT

So we need a separate pair of transport vlans per logical interface/per site? Can multiple logical interfaces use the same physical interface (I assume that should be ok).

Re: Fabric Extend over ISP (Hub-and-spoke)

Miguel-Angel_RO — Tue, 25 May 2021 16:58:13 GMT

That’s indeed the case. Multiple logical interfaces on one physical interface.

The MTU should be at least 1544.

Mig

Re: Fabric Extend over ISP (Hub-and-spoke)

bfaltys — Tue, 25 May 2021 20:21:36 GMT

Do you have a link to the guide you referenced? I’d like to look at that if it contains an example. I think I’m tracking on what to do, but a guide would be helpful just in case.

Re: Fabric Extend over ISP (Hub-and-spoke)

Miguel-Angel_RO — Tue, 25 May 2021 20:30:20 GMT

Here you are: https://www.extremenetworks.com/support/documentation/vsp-operating-system-software-voss-document-collections/

Mig

Re: Fabric Extend over ISP (Hub-and-spoke)

bfaltys — Tue, 25 May 2021 20:49:29 GMT

Thanks. I will look through that to see if/what I’m missing. I currently have an adjacency, but not seeing any MACs learned from the other switch.

TEST1:1#sh isis adj
************************************************************************************
Command Execution Time: Tue May 25 13:48:12 2021 UTC
************************************************************************************

====================================================================================================
                                ISIS Adjacencies
====================================================================================================
INTERFACE         L STATE        UPTIME PRI HOLDTIME SYSID             HOST-NAME         STATUS
----------------------------------------------------------------------------------------------------
2258-1/24         1 UP         00:12:08 127       27 0001.5000.0600    TEST2             ACTIVE

--------------------------------------------------------------------------------
1 out of 1 interfaces have formed an adjacency
--------------------------------------------------------------------------------

TEST1:1#sh vlan mac
TEST1:1#sh vlan mac-address-e
TEST1:1#sh vlan mac-address-entry
************************************************************************************
Command Execution Time: Tue May 25 13:48:18 2021 UTC
************************************************************************************

====================================================================================================
                                    Vlan Fdb
====================================================================================================
VLAN            MAC                                   SMLT
ID   STATUS     ADDRESS            INTERFACE          REMOTE   TUNNEL
----------------------------------------------------------------------------------------------------
10   learned    00:e0:4c:68:35:01 Port-1/14          false    -
10   self       f0:64:26:68:d9:00 Port-cpp           false    -
4051 self       00:01:50:00:05:00 Port-cpp           false    -
4051 self       00:01:50:00:06:00 2258-1/24          false    -
4051 self       12:50:05:ff:ff:ff Port-cpp           false    -
4051 self       12:50:06:ff:ff:ff 2258-1/24          false    -
4052 self       00:01:50:00:05:00 Port-cpp           false    -
4052 self       00:01:50:00:06:00 2258-1/24          false    -
4052 self       12:50:05:ff:ff:ff Port-cpp           false    -
4052 self       12:50:06:ff:ff:ff 2258-1/24          false    -

c: customer vid u: untagged-traffic

10 out of 10 entries in all fdb(s) displayed.

TEST1:1#

Re: Fabric Extend over ISP (Hub-and-spoke)

bfaltys — Tue, 25 May 2021 20:54:08 GMT

I didn’t have the i-sids configured. Doh! Looks like it is working!

Re: Fabric Extend over ISP (Hub-and-spoke)

Miguel-Angel_RO — Tue, 25 May 2021 20:56:19 GMT

Re: Fabric Extend over ISP (Hub-and-spoke)

bfaltys — Tue, 25 May 2021 21:09:01 GMT

If I add this to use a physical port that is already operating as an 802.1q trunk will it break the trunking? If it will I will have to configure a new interface and then move the link to the new one once everything is ready.

Re: Fabric Extend over ISP (Hub-and-spoke)

Miguel-Angel_RO — Tue, 25 May 2021 21:12:43 GMT

bfaltys,

I don’t think so but I can’t confirm.

I’m pretty sure you can add this to an existing dot1q interface without any issue.

You can easily test it on an unused interface. Tagg it and add a logical interface

Mig

Re: Fabric Extend over ISP (Hub-and-spoke)

bfaltys — Wed, 26 May 2021 01:59:16 GMT

Well, looks like it won’t work. At least I know and can plan accordingly!

Re: Fabric Extend over ISP (Hub-and-spoke)

bfaltys — Fri, 04 Jun 2021 19:50:25 GMT

Just wanted to share an update on this. We decided we didn’t want to use the 2 additional VLANs needed per site to map the BVLANs so I changed our uplink to the provider to a layer 3 interface. This allows us to route across the provider. Then set a loopback as a source IP. Create a logical interface that points to the remote loopback. You have to do this for each site, but can have multiple tunnels extending the fabric. Here is my configuration steps, test configs & some verification.

!!! CONFIGURATION STEPS !!!
-------------------------------------------------------------------
1. Set the prompt to help ID switches when in the CLI

prompt "SWITCH-NAME"

3. Configure loopback IP to be used as source/destination.

interface loopback <1-256>
ip address <IP/32>
exit

2. Configure basic ISIS & SPBM parameters.

spbm
router isis
sys-name SWITCH-NAME
system-id <ID>
manual-area 49.0001
spbm 1
spbm 1 nick-name <NICK-NAME>
spbm 1 b-vid 4051-4052 primary 4051
ip-tunnel-source-address <LOOPBACK-IP>
y
exit
vlan create 4051 name PBVID type spbm-bvlan
vlan create 4052 name SBVID type spbm-bvlan
router isis enable

3. Remove physical port from other vlans.

vlan members remove 1 1/24

4. Change physical port to routed port. Using VLAN 3000 as an example.
This VLAN is not a regular port VLAN. If you try to "create vlan 3000"
you'll get an error message.

interface g 1/24
brouter port 1/24 vlan 3000 subnet <INTERFACE-IP>/255.255.255.0
name "ISP-UPLINK"
no shut
exit

5. Configure logical interface with remote loopback as destination. A
logical interface will need to be configured per adjacency.

logical-intf isis 1 dest-ip <REMOTE-LOOPBACK> name <REMOTE-SWITCH-NAME>
isis
isis spbm 1
isis enable
exit

6. Create static route to remote loopback via layer 3 interface. Could use
OSPF to advertise the loopback IPs instead of a bunch of static routes.

ip route <REMOTE-LOOPBACK> 255.255.255.255 <REMOTE-ROUTED-INTERFACE-IP> weight 1
ip route <REMOTE-LOOPBACK> 255.255.255.255 <REMOTE-ROUTED-INTERFACE-IP> enable

7. Create test VLAN to see if MAC data is learned on remote switch

vlan create 99 name DATA type port-mstprstp 0
vlan i-sid 99 100099
vlan members add 99 1/2
interface g 1/2
no shut
exit

!!! SWITCH CONFIGURATIONS !!!
---------------------------------------------------------------------------------
Test switch 1

prompt "TEST1"

interface loopback 100
ip address 10.0.100.1/32
exit

spbm
router isis
sys-name TEST1
system-id 0001.5000.0500
manual-area 49.0001
spbm 1
spbm 1 nick-name 1.50.05
spbm 1 b-vid 4051-4052 primary 4051
ip-tunnel-source-address 10.0.100.1
y
exit
vlan create 4051 name PBVID type spbm-bvlan
vlan create 4052 name SBVID type spbm-bvlan
router isis enable

vlan members remove 1 1/24

interface g 1/24
brouter port 1/24 vlan 3000 subnet 10.30.30.1/255.255.255.0
no shut
exit

logical-intf isis 1 dest-ip 10.0.100.2 name TEST2
isis
isis spbm 1
isis enable
exit
logical-intf isis 2 dest-ip 10.0.100.3 name TEST3
isis
isis spbm 1
isis enable
exit
ip route 10.0.100.2 255.255.255.255 10.30.30.2 weight 1
ip route 10.0.100.2 255.255.255.255 10.30.30.2 enable
ip route 10.0.100.3 255.255.255.255 10.30.30.3 weight 1
ip route 10.0.100.3 255.255.255.255 10.30.30.3 enable

vlan create 99 name DATA type port-mstprstp 0
vlan i-sid 99 100099
vlan members add 99 1/2
interface g 1/2
no shut
exit
!!!CONNECT LINKRUNNER TO PORT 1/2 AND TURN ON
---------------------------------------------------------------------------------
Test switch 2

prompt "TEST2"

interface loopback 100
ip address 10.0.100.2/32
exit

spbm
router isis
sys-name TEST2
system-id 0001.5000.0600
manual-area 49.0001
spbm 1
spbm 1 nick-name 1.50.06
spbm 1 b-vid 4051-4052 primary 4051
ip-tunnel-source-address 10.0.100.2
y
exit
vlan create 4051 name PBVID type spbm-bvlan
vlan create 4052 name SBVID type spbm-bvlan
router isis enable

vlan members remove 1 1/24

interface g 1/24
brouter port 1/24 vlan 3000 subnet 10.30.30.2/255.255.255.0
no shut
exit

logical-intf isis 1 dest-ip 10.0.100.1 name TEST1
isis
isis spbm 1
isis enable
exit
logical-intf isis 2 dest-ip 10.0.100.3 name TEST3
isis
isis spbm 1
isis enable
exit
ip route 10.0.100.1 255.255.255.255 10.30.30.1 weight 1
ip route 10.0.100.1 255.255.255.255 10.30.30.1 enable
ip route 10.0.100.3 255.255.255.255 10.30.30.3 weight 1
ip route 10.0.100.3 255.255.255.255 10.30.30.3 enable

vlan create 99 name DATA type port-mstprstp 0
vlan i-sid 99 100099

---------------------------------------------------------------------------------
Test switch 3

prompt "TEST3"

interface loopback 100
ip address 10.0.100.3/32
exit

spbm
router isis
sys-name TEST1
system-id 0001.5000.0700
manual-area 49.0001
spbm 1
spbm 1 nick-name 1.50.07
spbm 1 b-vid 4051-4052 primary 4051
ip-tunnel-source-address 10.0.100.3
y
exit
vlan create 4051 name PBVID type spbm-bvlan
vlan create 4052 name SBVID type spbm-bvlan
router isis enable

vlan members remove 1 1/24

interface g 1/24
brouter port 1/24 vlan 3000 subnet 10.30.30.3/255.255.255.0
no shut
exit

logical-intf isis 1 dest-ip 10.0.100.1 name TEST1
isis
isis spbm 1
isis enable
exit
logical-intf isis 2 dest-ip 10.0.100.2 name TEST2
isis
isis spbm 1
isis enable
exit
ip route 10.0.100.1 255.255.255.255 10.30.30.1 weight 1
ip route 10.0.100.1 255.255.255.255 10.30.30.1 enable
ip route 10.0.100.2 255.255.255.255 10.30.30.2 weight 1
ip route 10.0.100.2 255.255.255.255 10.30.30.2 enable

vlan create 99 name DATA type port-mstprstp 0
vlan i-sid 99 100099
---------------------------------------------------------------------------------

!!! VERIFY ADJACENCIES & MAC LEARNING !!!
---------------------------------------------------------------------------------
On switch TEST1, verify ISIS adjacencies. Then connect link runner to access
port 1/2 in vlan 99 (i-sid 100099).

TEST1:1#sh isis adj
************************************************************************************
Command Execution Time: Thu Jun 03 19:50:47 2021 UTC
************************************************************************************

===============================================================================
                                ISIS Adjacencies
===============================================================================
INTERFACE         L STATE        UPTIME PRI HOLDTIME SYSID             HOST-NAME         STATUS
----------------------------------------------------------------------------------------------------
TEST2             1 UP         00:38:36 127       22 0001.5000.0600    TEST2             ACTIVE
TEST3             1 UP         00:24:59 127       20 0001.5000.0700    TEST3             ACTIVE

--------------------------------------------------------------------------------
2 out of 2 interfaces have formed an adjacency
--------------------------------------------------------------------------------

TEST1:1#

TEST1:1#show vlan mac-address-entry port 1/2
************************************************************************************
Command Execution Time: Thu Jun 03 19:52:10 2021 UTC
************************************************************************************

===============================================================================
                                    Vlan Fdb
===============================================================================
VLAN            MAC                                   SMLT
ID   STATUS     ADDRESS            INTERFACE          REMOTE   TUNNEL
----------------------------------------------------------------------------------------------------
99   learned    00:c0:17:b7:90:d2 Port-1/2           false    -

c: customer vid u: untagged-traffic

1 out of 17 entries in all fdb(s) displayed.

TEST1:1#

---------------------------------------------------------------------------------
On switch TEST2, verify adjacencies and remote MAC learning.

TEST2:1#sh isis adj
************************************************************************************
Command Execution Time: Thu Jun 03 19:54:07 2021 UTC
************************************************************************************

===============================================================================
                                ISIS Adjacencies
===============================================================================
INTERFACE         L STATE        UPTIME PRI HOLDTIME SYSID             HOST-NAME         STATUS
----------------------------------------------------------------------------------------------------
TEST1             1 UP         00:40:59 127       19 0001.5000.0500    TEST1             ACTIVE
TEST3             1 UP         00:12:30 127       22 0001.5000.0700    TEST3             ACTIVE

TEST2:1#sh vlan mac-address-entry
************************************************************************************
Command Execution Time: Thu Jun 03 19:54:57 2021 UTC
************************************************************************************

---------------------------------------------------------------------------------
On switch TEST3, verify adjacencies and remote MAC learning.

TEST3:1#sh isis adj
************************************************************************************
Command Execution Time: Thu Jun 03 19:58:22 2021 UTC
************************************************************************************

===============================================================================
                                ISIS Adjacencies
===============================================================================
INTERFACE         L STATE        UPTIME PRI HOLDTIME SYSID             HOST-NAME         STATUS
----------------------------------------------------------------------------------------------------
TEST1             1 UP         00:28:58 127       24 0001.5000.0500    TEST1             ACTIVE
TEST2             1 UP         00:14:07 127       23 0001.5000.0600    TEST2             ACTIVE

TEST3:1#sh vlan mac-address-entry
************************************************************************************
Command Execution Time: Thu Jun 03 19:58:27 2021 UTC
************************************************************************************