topic RE: Routing between virtual routers in ExtremeSwitching (EXOS/Switch Engine)

Routing between virtual routers

Frank — Tue, 25 Mar 2014 16:33:00 GMT

I'd like to revisit the question of "how do I route between virtual routers". I'm in a managed datacenter environment and have 8806s, 480s, 460s. all running XOS 15.3 or higher.

Scenario: I have a dozen or so corporate-internal VLANs that are all connected to let's say "VR-Corp". Now, the problem is that I have a handful of colocated customers with their internal networks (private IP space) that we need to manage, so two or three of our corporate VLANs have to somehow get access to the customer VLANs.

I do not want to add those customer VLANs to our corporate VR. I'm not a big fan of ACLs that aren't straightforward, simple, and easily maintainable. VPN access to the customer is usually not an option, either.
I would much rather add all those customer VLANs to a "VR-Cust" and somehow route between the to VRs - that approach makes for a much simpler configuration on the respective VRs. At least I won't have to worry about routing protocols - just good old fashioned static routes will do just fine here 🙂

From previous discussions I do understand that I cannot do that "within" the switch. However, my 8806s do have a 48-port Ethernet blade, and my idea was to create one VLAN on each VR (different tags) that are the same IP network, assign one Ethernet port to each VLAN, and just patch them together with a short cable.

This, however, does not seem to work. The ports are up, but I can only ping the IP address that's on the VR that I set the context to, and not the other. The ports are up, the VRs are up, the VLANs are up, and they're still invisible to each other. I would assume that if it doesn't work over Ethernet, it won't work over fiber either.

I don't quite understand why it doesn't work - technically I'm leaving the switch out one port and come back in through another port.

If everything else fails, I can of course introduce my 480 into the mix and have it be the "router between the VRs" (or rather: the cross-connect VLANs)", but I would find that a somewhat less-than-elegant solution (to a less-than-elegant problem/requirement).

Thanks for your help!

Frank

RE: Routing between virtual routers

Paul_Russo — Tue, 25 Mar 2014 18:44:00 GMT

Good Morning Frank Thanks for the question. If I read your post correctly you have a two port VLAN (I will call it interconnect) that is in both VRs is that correct?

I assume that you are routing between the customer/corporate VLANs to the interconnect VLAN?

I will do some checking internally but wanted to make sure that the interconnect VLAN was set as the routed segment.

Thanks
P

RE: Routing between virtual routers

Frank — Tue, 25 Mar 2014 18:44:00 GMT

Correct. The idea would be "route customer to support vlan nexthop interconnect vlan" and the corresponding "route support technician vlan to customer vlan"

Basically the following setup. I'm not even at the routing part, because vr-test1 can't even ping 172.18.18.252 (see very bottom)

I'm starting to get a little out of my depth here, but I think what's killing my plan is that (got this from somewhere in the Concepts or Commands Guide PDF) ALL the VRs (and VRFs etc.) on ONE switch have the SAME MAC address - which also can't be manually configured on a per-VR/VLAN/port basis. But as I said, there's a good chance that I'm not sure what I'm talking about 🙂

"Of course" the end-goal if we get this to work is to replicate it all on our second 8806, including setting up vrrp, but right now I've left this out because that'll just complicate matters. I think...

Thank you for your fast reply!

------------------------------------------------------------------
create vr vr-test1
vr "vr-test1"
configure "vr-test1" add ports 5:24

create vlan vlan_test1
configure vlan "vlan_test1" tag 2000
configure "vlan_test1" ipaddress 172.18.18.1/24

create vr vr-test2
vr "vr-test2"
configure "vr-test2" add ports 5:48

create vlan vlan_test2
configure vlan "vlan_test2" tag 2002
configure "vlan_test2" ipaddress 172.18.18.252/24

enable loopback-mode vlan_test1 (or disable, doesn't make a difference)
enable loopback-mode vlan_test2 (----------^^---------)

configure vlan_test1 add ports 5:24 untagged
configure vlan_test2 add ports 5:48 untagged
----------------------------------------------------------------
Physical Ethernet cable connects 5:24 and 5:48.

----------------------------------------------------------------

Core-2 # vr vr-test1
(vr vr-test1) Core-2.7 # ping 172.18.18.252
Ping(ICMP) 172.18.18.252: 4 packets, 8 data bytes, interval 1 second(s).

--- 172.18.18.252 ping statistics ---
4 packets transmitted, 0 packets received, 100% loss
round-trip min/avg/max = 0/0/0 ms

(P.S.: Sorry, got confused with "comment" vs "reply". Hope "comment" works...)

RE: Routing between virtual routers

Paul_Russo — Wed, 26 Mar 2014 03:26:00 GMT

Hey Frank

My suspicion was correct. The issue is that the switch has one MAC for all VRs.

Here's some information from an internal post

At L2, a simple cable between the two VRs will do the job. Disable learning on the ports will help.

At L3, we need an external device to go from one VR to another. And because we have a unique Mac for the whole system, you may need 2 external devices... Hopefully, VRF leaking will come one day (no idea of possible limitations). In the meantime, you can try to trick the system to enable such feature using a cable to directly connect two ports in two VRs, and using VRRP to generate a different mac.

I would recommend going to a 460/480 versus trying to do it with VRRP.

Let me know if there is anything else I can help with.
P

RE: Routing between virtual routers

Sumit_Tokle — Wed, 26 Mar 2014 11:27:00 GMT

We do support below,

PC1-----switch------PC2

PC1:100.1.1.2/24
PC2:200.1.1.2/24

Configuration on switch:
create virtual-router vrf1 type vrf "VR-Default"virtual-router vrf1
configure vr "VR-Default" delete ports 7:8 7:9
create vlan v1
configure vlan v1 add ports 7:8
configure v1 ipaddress 100.1.1.1/24
en ipf

create virtual-router vrf2 type vrf "VR-Default"
virtual-router vrf2
create vlan v2
configure v2 add ports 7:9
configure v2 ipaddress 200.1.1.1/24
en ipf

VRF forwarding will not be done. So am adding an static route in vrf1 as

virtual-router vrf1

configure iproute add 200.1.1.0/24 200.1.1.2 vlan "v2"

RE: Routing between virtual routers

Frank — Wed, 26 Mar 2014 11:27:00 GMT

That's an interesting approach. I'm surprised that works; I didn't even think of trying that!

My "main" VLAN has to be in a full VR (routing protocols), I think, and as I would have to add and protect multiple VLANs in the VRF I'll have to see if I can set simple ACLs.

I'll play with that - thank you much!

RE: Routing between virtual routers

Frank — Wed, 26 Mar 2014 16:10:00 GMT

Paul,

In the meantime, you can try to trick the system to enable such feature using a cable to directly connect two ports in two VRs, and using VRRP to generate a different mac.

I actually started out with both of my connected 8806s having both VLANs defined, both 8806s having the patch cable, and I configured VRRP on both the VRs - problem was that I was getting a bunch of "ignoring lower priority VRRP advertsising" notifications (quoted from memory) that I'm not used to get on other VRRPs, but that could've been because I didn't explicitely define "IP tracking", and it might have broadcasted to the entire network, seeing IPs on duplicate MACs and all that.

I might revisit that, though, "just because!" (and I'm 'tenacious'). I'll also give Sumit Tokle's idea a shot, but one VLAN has to be in a full VR (because of ospf).

Heck, if all else fails, we do have abusable 480s 😉

Thank you very much - I'll try things out and reply in here how it went.

Frank

RE: Routing between virtual routers

Frank — Wed, 02 Apr 2014 21:53:00 GMT

Well, so far, it's "a learning experience" 🙂

Generating a new MAC via the VRID: doesn't work too well. Yes, I have a made-up MAC, and yes, it's unique, but the switch doesn't want to get tricked. I presume packets stays on L2 and don't want to get forced to L3, because the switch is just too smart.

Sumit's approach: I can't guarantee the presence of "200.1.1.2" on that network or on that switch, so I can't use it, and I can't use the VR's IP because I run into MAC issues again. Or at least I think that's why that failed.

I ended up using our 480s to act as an external router between virtual routers. So I have one network/vlan "vlan_test1" in "VR-Test1" (172.18.0.0/24) and a vlan "vlan_test2" in VR-Test2" (172.18.1.0/24), tag them both to the 480s.
On the 480 I have "VR-XCHANGE", with both vlans (test1 and test2)", ipforwarding enabled.

Since we're talking about two 8800s and two 480s (yay, VRRP!!!), I did accidentally assign the same VRID to the VR on the 8800s and the VR on the 480s. Bad bad bad idea, because all of a sudden they all thought they needed to be able to be redundant, but they saw different virtual IPs (Of course! One for the 480s, one for the 8800s) and freaked out. Changed the VRID on the 480s and voila! all is well!

Now I know what else VRIDs do, other than change the virtual MAC  I think, Paul, you mentioned that in another thread about VRRPs - thanks, saved me today.

It would be oh-so-nice if I could've just assigned MACs to ports and used a patch-cable, but I understand the design history/decision.

Thanks for everyone's input - and if I ever find another way to skin that cat, I'll be back in this thread 🙂

RE: Routing between virtual routers

Paul_Russo — Wed, 02 Apr 2014 22:24:00 GMT

Hey Frank

Thanks for sharing this with the community as I am sure that it will help someone else down the road.

P