S Series VXLAN LAB setup

  • 0
  • 1
  • Question
  • Updated 11 months ago
  • Answered
  • (Edited)
Hi

I am busy playing with VXLAN between two S series switches in my LAB and have some problems.



The two S series switches connect to each other across a L3 routed network.
I have configured the VXLAN tunnel and that seems to be working fine.

VXLAN Setup for "SSA One":

 interface tun.0.1  tunnel mode vxlan l2 tbp.0.1
  tunnel source 172.16.1.1
  no shutdown
  exit

set tunnel logical-switch create name Switch1
set tunnel map logical-switch Switch1 keyword 111 vlan 100
set tunnel remote-vtep logical-switch Switch1 ip-address 172.16.1.2

set vlan egress 100 tbp.0.1 tagged
set vlan egress 100 ge.2.48 untagged (User Port)

VXLAN Setup for "SSA Two"

 interface tun.0.1  tunnel mode vxlan l2 tbp.0.1
  tunnel source 172.16.1.2
  no shutdown
  exit

set tunnel logical-switch create name Switch1
set tunnel map logical-switch Switch1 keyword 111 vlan 100
set tunnel remote-vtep logical-switch Switch1 ip-address 172.16.1.1

set vlan egress 100 tbp.0.1 tagged
set vlan egress 100 tg.1.24 untagged (User Port)

With the above configuration, devices connected to both S series switches in vlan 100 can ping each other.

-----------------------------------------------------------------------------------------

For the next part of my lab I would like to test routing from the remote site (SSA Two) to a network that is connected to "SSA one"

For this I added another vlan on "SSA one" and created a routed interface.

For users in vlan 100 (the VXLAN vlan) I have specified 192.168.100.1 (Vlan 100 interface on "SSA One") as the default gateway.
Clients connected the "SSA one" in vlan 100 can ping any device in vlan 10 and any device in vlan 10 can ping any device connected to "SSA one" in vlan 100.

The problem I have is with the remote clients.
The Remote clients connected to "SSA Two" in vlan 100 can ping any other device in vlan 100 (Both SSA one and SSA Two). These clients can also ping the Vlan 10 interface of "SSA One" (192.168.10.211), but they can not ping any other device in vlan 10.
The Vlan 10 devices can not ping any of the remote vlan 100 (SSA two) clients.

Why is this?

If I extend vlan 100 from "SSA One" to an external router that routes traffic for both Vlan 100 and Vlan 10 then every client can reach every other client.
Only when I use one of the terminating VXLAN SSA's for routing can the remote client not route out.

Hope this makes sense.... 

Regards
Photo of Andre Brits Kannemeyer

Andre Brits Kannemeyer

  • 5,010 Points 5k badge 2x thumb

Posted 2 years ago

  • 0
  • 1
Photo of Ronald Dvorak

Ronald Dvorak, Embassador

  • 45,120 Points 20k badge 2x thumb
Could you try a default route instead of the network route on the bottom router
or
set the bottom clients default gateway to .211 instead of .1

and try again - I wonder whether that has any effect.....
Photo of Erik Auerswald

Erik Auerswald, Embassador

  • 12,782 Points 10k badge 2x thumb
Hi Andre,

what you are describing seems related to the usual case with VXLAN Layer 2 gateway functionality, where you need an external router to route between VXLANs.

Your setup is not the usual VXLAN data center use case, but more of a traditional Layer 2 point-to-point tunnel setup.

I would expect connectivity between 192.168.100.111 and 192.168.10.10 if both VLANs are routed on SSA One and both PCs use SSA One as default gateway. It would be interesting to know if adding the VXLAN tunnel breaks this.

The case for connectivity between 192.168.100.112 and 192.168.10.10 is unusual for VXLAN setups, because SSA One would need to decapsulate the VXLAN packet and then run the resulting frame through VLAN/SVI routing. The usual method is to route to the target VXLAN on the encapsulating switch, if you have a Layer 3 VXLAN gateway.

Does your setup work without an external router if you use a GRE L2 tunnel between the SSAs?

This is an interesting problem. :-)

Best regards,
Erik
Photo of Andre Brits Kannemeyer

Andre Brits Kannemeyer

  • 5,010 Points 5k badge 2x thumb
Ok so I have simplified the diagram:


Summary:

User1
Vlan 100
Default Gateway = 192.168.100.1

User2
Vlan 100
Default Gateway = 192.168.100.1

User3
Vlan 10
Default Gateway = 192.168.10.211


Ping results:
User1
Ping to Default Gateway (192.168.100.1) = Successful
Ping to Vlan 10 Router Interface (192.168.10.211) = Successful
Ping to User 2 = Successful
Ping to User 3 = Successful

User2
Ping to Default Gateway (192.168.100.1) = Successful
Ping to Vlan 10 Router Interface (192.168.10.211) = Successful
Ping to User 1 = Successful
Ping to User 3 = Fail

User3
Ping to Default Gateway (192.168.10.211) = Successful
Ping to Vlan 100 Router Interface (192.168.100.1) = Successful
Ping to User 1 = Successful
Ping to User 2 = Fail
 

As previously mentioned when I move the routing function from the SSA to a external router (routing between vlan 10 and 100) all devices can ping all other devices.

I would have presumed that if User1 can access vlan 10 devices then and other device in vlan 100 with the same gateway would also be able to get to vlan 10.

I would have presumed that the SSA acts like a VXLAN L3 gateway when you terminate a tunnel on it.

This is my firt time in the ring the VXLAN so please correct me if I am wrong.
I will test the same setup with a GRE tunnel and report back.

Thx
Andre
 
(Edited)
Photo of Andre Brits Kannemeyer

Andre Brits Kannemeyer

  • 5,010 Points 5k badge 2x thumb
Ok so I have tested the same setup but this time useing just GRE tunnels

SSA one:
interface tun.0.1  tunnel destination 172.16.1.2
  tunnel mode gre l2 tbp.0.1
  tunnel source 172.16.1.1
  no shutdown
  exit
set spantree portadmin tbp.0.1 disable
set spantree portadmin tg.1.1 disable
set vlan egress 100 tbp.0.1 tagged

SSA Two:
interface tun.0.1  tunnel destination 172.16.1.1
  tunnel mode gre l2 tbp.0.1
  tunnel source 172.16.1.2
  no shutdown
  exit
set spantree portadmin tbp.0.1 disable
set spantree portadmin tg.1.1 disable
set vlan egress 100 tbp.0.1 tagged


All three users every other user and Router Interface.

So the question is why does this not work with VXLAN.

I am new to VXLAN but perhaps this operates in the same way as TRILL where you can not use the same switch to terminate trill and route....
Photo of Erik Auerswald

Erik Auerswald, Embassador

  • 12,782 Points 10k badge 2x thumb
I'll try this out as well. So far I have replicated the GRE L2 tunnel setup, which works as intended, as you wrote above.
Photo of Erik Auerswald

Erik Auerswald, Embassador

  • 12,782 Points 10k badge 2x thumb
Hm, I have changed the working GRE L2 setup to use VXLAN instead, and it <EDIT> still works fine stopped working </EDIT>. Id o not see the problem in your config.

The PC connected to VLAN 100 on SSA One can ping both the VLAN 100 PC on SSA Two (L2 over VXLAN tunnel between SSA One and Two) and the VLAN 10 PC on SSA One (L3 forwarding on SSA One).

I have enabled jumbo frames to support a 1500 byte MTU over the VXLAN tunnel, but this is not needed for basic ping connectivity.

The relevant configuration is as follows (I have changed some IP addresses and use different ports):

SSA One
configure terminal
!
 interface loop.0.1
  ip address 172.16.1.1 255.255.255.255 primary
  no shutdown
  exit
 interface vlan.0.10
  ip address 192.168.10.1 255.255.255.0 primary
  no shutdown
  exit
 interface vlan.0.100
  ip address 192.168.100.1 255.255.255.0 primary
  no shutdown
  exit
 interface vlan.0.999
  ip address 10.1.1.0 255.255.255.254 primary
  no shutdown
  exit
 interface tun.0.1
  tunnel mode vxlan l2 tbp.0.1
  tunnel source 172.16.1.1
  no shutdown
  exit
!
# Static routes configured on routed interfaces
 ip route 172.16.2.2/32 10.1.1.1 interface vlan.0.999 1
!
exit
# port
set port disable lag.0.1-62
set port disable tbp.0.2-62
set port disable ge.1.1-4,8-48
set port disable tg.1.1-4
set port jumbo enable ge.1.6
set port vlan ge.1.5 100
set port vlan ge.1.6 999
set port vlan ge.1.7 10
# spantree
set spantree portadmin tbp.0.1 disable
# tunnel
set tunnel logical-switch create name VXLAN_LAB
set tunnel map logical-switch VXLAN_LAB keyword 1000 vlan 100
set tunnel remote-vtep logical-switch VXLAN_LAB ip-address 172.16.2.2
# vlan
set vlan create 10,100,999
set vlan name 10 LOCAL_CLIENTS
set vlan name 100 DISTRIBUTED_CLIENTS
set vlan name 999 WAN
clear vlan egress 1 tbp.0.1;ge.1.5-7
set vlan egress 10 ge.1.7 untagged
set vlan egress 100 tbp.0.1 tagged
set vlan egress 100 ge.1.5 untagged
set vlan egress 999 ge.1.6 untagged
SSA Two
!
 interface loop.0.1
  ip address 172.16.2.2 255.255.255.255 primary
  no shutdown
  exit
 interface vlan.0.999
  ip address 10.1.1.1 255.255.255.254 primary
  no shutdown
  exit
 interface tun.0.1
  tunnel mode vxlan l2 tbp.0.1
  tunnel source 172.16.2.2
  no shutdown
  exit
!
# Static routes configured on routed interfaces
 ip route 172.16.1.1/32 10.1.1.0 interface vlan.0.999 1
!
exit
# port
set port disable lag.0.1-62
set port disable tbp.0.2-62
set port disable ge.1.1-4,7-48
set port disable tg.1.1-4
set port jumbo enable ge.1.6
set port vlan ge.1.5 100
set port vlan ge.1.6 999
# spantree
set spantree portadmin tbp.0.1 disable
# tunnel
set tunnel logical-switch create name VXLAN_LAB
set tunnel map logical-switch VXLAN_LAB keyword 1000 vlan 100
set tunnel remote-vtep logical-switch VXLAN_LAB ip-address 172.16.1.1
# vlan
set vlan create 100,999
set vlan name 100 DISTRIBUTED_CLIENTS
set vlan name 999 WAN
clear vlan egress 1 tbp.0.1;ge.1.5-6
set vlan egress 100 tbp.0.1 tagged
set vlan egress 100 ge.1.5 untagged
set vlan egress 999 ge.1.6 untagged
(Edited)
Photo of Erik Auerswald

Erik Auerswald, Embassador

  • 12,782 Points 10k badge 2x thumb
One difference is that I do not have an SVI for VLAN 100 on SSA Two...

Edit: Just tested this, it does not matter.
(Edited)
Photo of Erik Auerswald

Erik Auerswald, Embassador

  • 12,782 Points 10k badge 2x thumb
To recap my tests: with VXLAN, layer 2 connectivity is working. All three test PCs can reach all other PCs and the SVIs in the same VLAN, including the VXLAN extension for the VLAN. But there is no connectivity between User3 and User2. Using GRE L2 tunneling instead of VXLAN L2 tunneling works.

That differs from the initial description by Andre, where User1 did not have connectivity to User3.
Photo of Erik Auerswald

Erik Auerswald, Embassador

  • 12,782 Points 10k badge 2x thumb
I achieve full connectivity by adding VLAN 10 to the VXLAN overlay:

SSA One
set tunnel logical-switch create name VXLAN_LAB
set tunnel map logical-switch VXLAN_LAB keyword 100 vlan 100
set tunnel logical-switch create name VXLAN_10
set tunnel map logical-switch VXLAN_10 keyword 10 vlan 10
set tunnel remote-vtep logical-switch VXLAN_LAB ip-address 172.16.2.2
set tunnel remote-vtep logical-switch VXLAN_10 ip-address 172.16.2.2
SSA Two
set tunnel logical-switch create name VXLAN_LAB
set tunnel map logical-switch VXLAN_LAB keyword 100 vlan 100
set tunnel logical-switch create name VXLAN_10
set tunnel map logical-switch VXLAN_10 keyword 10 vlan 10
set tunnel remote-vtep logical-switch VXLAN_LAB ip-address 172.16.1.1
set tunnel remote-vtep logical-switch VXLAN_10 ip-address 172.16.1.1
That is unexpected. I would call this a bug. :-(
Photo of Andre Brits Kannemeyer

Andre Brits Kannemeyer

  • 5,010 Points 5k badge 2x thumb
Hi Erik

Thank you for your efforts here.
I found exactly the same results.

Only once I add vlan 10 to the Tunnel config can every device see every other device.


SSA One
set tunnel logical-switch create name Switch2
set tunnel map logical-switch Switch2 keyword 110 vlan 10
set tunnel logical-switch create name Switch1
set tunnel map logical-switch Switch1 keyword 111 vlan 100
set tunnel remote-vtep logical-switch Switch2 ip-address 172.16.1.2
set tunnel remote-vtep logical-switch Switch1 ip-address 172.16.1.2

SSA two
set tunnel logical-switch create name Switch2
set tunnel map logical-switch Switch2 keyword 110 vlan 10
set tunnel logical-switch create name Switch1
set tunnel map logical-switch Switch1 keyword 111 vlan 100
set tunnel remote-vtep logical-switch Switch2 ip-address 172.16.1.1
set tunnel remote-vtep logical-switch Switch1 ip-address 172.16.1.1
(Edited)
Photo of Andre Brits Kannemeyer

Andre Brits Kannemeyer

  • 5,010 Points 5k badge 2x thumb
The reason for my LAB is that we have a client with two Data Centers. 
I would like to extended some of the server vlans across the two datacentres.
The reason for testing the routing to a send outside vlan is that the two DC's connect to a very big OSPF cloud, so these VXLAN vlans must be able to route to the clients in the OSPF cloud....
Photo of Erik Auerswald

Erik Auerswald, Embassador

  • 12,782 Points 10k badge 2x thumb
Hi Andre,

have you considered using GRE L2 tunneling instead? As I see it, VXLAN can be used for DCI, but other methods exist as well and might be better tested.

Br,
Erik
Photo of Andre Brits Kannemeyer

Andre Brits Kannemeyer

  • 5,010 Points 5k badge 2x thumb
I have thx.... wanted to use the new stuff ;)
Lets wait and see what we get from the extreme guys on this....
Photo of Drew C.

Drew C., Community Manager

  • 37,336 Points 20k badge 2x thumb
Hi Andre,
Since this is suspected to be a bug, would you please open a case with GTAC so it can be properly reviewed?
It will also be helpful to include a link to this thread in the case details.
Photo of Andre Brits Kannemeyer

Andre Brits Kannemeyer

  • 5,010 Points 5k badge 2x thumb