Question

Can't Reach Interface After Changing VRF


Userlevel 5
Hi,

There is probably a very simple answer to this question, but can't think what it is?

Very high level I have the following configuration:

code:
interface vlan 1999
description "Server Switch Management"
exit

config t
rbridge 11
interface ve 1999
ip address 172.24.110.11/24
ip route 0.0.0.0/0 172.24.110.254

vcs virtual ip address 172.24.110.10/24 inband interface ve 1999

configure
interface port-channel 10
switchport
switchport mode trunk
switchport trunk allowed vlan add 1999
switchport trunk tag native-vlan
no shutdown

int te 11/0/1
channel-group 10 mode active type standard
lacp default-up
int te 12/0/1
channel-group 10 mode active type standard
lacp default-up


This worked find and I could reach both the virtual IP address (172.24.110.10) and the Vlan 1999 interface (172.24.110.10).

The logical chassis has no other L3 addresses as is primarily just being used at L2.

The switch was on a version 6 code, and had been trying to upgrade it to version 7 but just kept erroring that it couldn't reach the SCP / FTP server even though I could ping it.

It seems in version 6 your unable to select the VRF to use when doing the firmware download, so it is defaulting I believe to mgmt-vrf.

So what I did is add the following command to ve 1999

code:
vrf forwarding mgmt-vrf


Since I did that I lost contact to switch. I did anticipate that could happen although I'm not sure why it did?

The other end of the portchannel has the IP address 172.24.110.254, and I would have expected to still been able to reach the switch from the local subnet?

Although the VRF has changed I would expect the VLAN to automatically just reside to the same VRF i.e. just moved from default to management.

Hence where I am stuck, perhaps I'm missing another command?

Many thanks in advance.

10 replies

Userlevel 2
Martin,

I believe the issue is that anytime you add or change a VRF all L3 configuration is removed from an interface.

Therefore if you were accessing the device using the IP for VE 1999 I would have expected your connection to be terminated as the IP address for this VE should no longer be configured. I would suggest to console into the device and reconfigure the IP address on the VE.

Example:

code:
VDX1# show run rb 1 int ve 1000
rbridge-id 1
interface Ve 1000
ip proxy-arp
ip address 10.10.10.1/24
no shutdown
!
!
Static-Lab-SM08_VDX1# conf t
Entering configuration mode terminal
Static-Lab-SM08_VDX1(config)# rb 1
Static-Lab-SM08_VDX1(config-rbridge-id-1)# int ve 1000
Static-Lab-SM08_VDX1(config-rbridge-Ve-1000)# vrf forwarding mgmt-vrf
Static-Lab-SM08_VDX1(config-rbridge-Ve-1000)# end
Static-Lab-SM08_VDX1# show run rb 1 int ve 1000
rbridge-id 1
interface Ve 1000
vrf forwarding mgmt-vrf
no shutdown


Also, regarding your SCP/FTP issues. If you can login as your root account you can attempt to manually connect to the FTP server to verify connectivity, username/password, and file path using standard LInux/CLI FTP commands.

I hope this helps resolve your issue.

Mike Morey
Principal Technical Support Engineer
Userlevel 5
Hi Mike,

Thanks for responding. Got some advise from an Extreme Engineer whom said the same thing, which it was.

Added the IP address back in, but then had another issue with the default gateway. For anyone reading this it made sense to put the route under the vrf, in this case mgmt-vrf, but it would not except the 'ip route' command.

Turns out it needs to be added another level down under the address-family, see below:

code:
vrf mgmt-vrf
address-family ipv4 unicast
ip route 0.0.0.0/0 172.24.110.254


The problem I have now is that I can access SCP / FTP server, which in this case is ExtremeManagement, and see to have the folder structure, that being /root/, so my directory string needs to be /firmware/images



When trying to upgrade I have tried all the below, none which work?

code:
firmware download logical-chassis scp rbridge-id all coldboot user root password xxxx host x.x.x.x directory /firmware/nos7.0.2b
firmware download logical-chassis scp rbridge-id all coldboot user root password xxxx host x.x.x.x directory / file nos7.0.2b
firmware download logical-chassis scp rbridge-id all coldboot user root password xxxx host x.x.x.x directory /firmware/images/nos7.0.2b
firmware download logical-chassis ftp rbridge-id all coldboot user anonymours password xxxx host x.x.x.x directory /firmware/images/nos7.0.2b
firmware download logical-chassis scp rbridge-id all coldboot user root password xxxx host x.x.x.x directory /root/firmware/images/
firmware download logical-chassis scp rbridge-id all coldboot user root password xxxx host x.x.x.x directory /root/firmware/images/nos7.0.2b
firmware download logical-chassis ftp rbridge-id all coldboot user anonymours password xxxx host x.x.x.x directory /tftpboot/firmware/images/
firmware download logical-chassis scp rbridge-id all coldboot user root password xxxx host x.x.x.x directory firmware/images/
firmware download logical-chassis scp rbridge-id all coldboot user root password xxxx host x.x.x.x directory firmware/images/nos7.0.2b


code:
 Col-xxx-VSP-Sw11# ping x.x.x.x vrf mgmt-vrf
Type Control-c to abort
PING x.x.x.x (x.x.x.x): 56 data bytes
64 bytes from x.x.x.x: icmp_seq=0 ttl=60 time=3.126 ms
64 bytes from x.x.x.x: icmp_seq=1 ttl=60 time=2.330 ms
64 bytes from x.x.x.x: icmp_seq=2 ttl=60 time=3.492 ms
64 bytes from x.x.x.x: icmp_seq=3 ttl=60 time=4.323 ms
64 bytes from x.x.x.x: icmp_seq=4 ttl=60 time=3.262 ms
--- x.x.x.xping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max/stddev = 2.330/3.307/4.323/0.641 ms


code:
Col-xxx-VSP-Sw11# show version

Network Operating System Software
Network Operating System Version: 6.0.2
Copyright (c) 1995-2015 Brocade Communications Systems, Inc.
Firmware name: 6.0.2h
Build Time: 22:04:55 Apr 12, 2018
Install Time: 19:35:32 Feb 8, 2019
Kernel: 2.6.34.6

BootProm: 1.0.1
Control Processor: e500mc with 4096 MB of memory

Slot Name Primary/Secondary Versions Status
---------------------------------------------------------------------------
SW/0 NOS 6.0.2h ACTIVE*
6.0.2h
SW/1 NOS 6.0.2h STANDBY
6.0.2h


Can you see anything that I am missing or got incorrect?

Many thanks.
Userlevel 2
This looks to be a path issue.

code:
firmware download logical-chassis scp rbridge-id all coldboot user root password xxxx host x.x.x.x directory /root/firmware/images/nos7.0.2b


Another user had similar issues here:

https://community.extremenetworks.com/data-center-slx-vdx-mlx-ces-232983/upgrading-vdx-over-scp-7822679

Can you collect the same output from your SSH server and provide it?

code:
sw0# ssh x.x.x.x -l root vrf mgmt-vrf


Once you connected to your SCP server, run the following and paste it back here:

code:
$ ls -R /root/firmware/images/nos7.0.2b | grep ":$" | sed -e 's/:$//' -e 's/[^-][^\/]*\//--/g' -e 's/^/  /' -e 's/-/|/' | head -20
Userlevel 5
Hi Truyen,

Thanks for getting back, here is the results:

root@NetSightCOL01.abc.co.uk:~$ pwd
/root

root@NetSightCOL01.abc.co.uk:~$ ls -R /root/firmware/images/nos7.0.2b | grep ":$" | sed -e 's/:$//' -e 's/[^-][^\/]*\//--/g' -e 's/^/ /' -e 's/-/|/' | head -20
|-----nos7.0.2b
|-------common
|---------BP
|-------SWBD1000
|-------SWBD1001
|-------SWBD1002
|-------SWBD1003
|-------SWBD1004
|-------SWBD1005
|-------SWBD1006
|-------SWBD1007
|-------SWBD1008
|-------SWBD1009
|-------SWBD1010
|-------SWBD1011
|-------SWBD131
|-------SWBD137
|-------SWBD138
|-------SWBD151
|-------SWBD153

Wondering if its a permission thing in ExtremeManagement, going to take a look at that next.


In the past I've had to add the -d to the Netsight nstftpd.cfg file to be able to use TFTP, although it does potentially make it less secure.

Will perhaps try a different SCP / SFTP server?

Thanks,

Martin
Userlevel 5
Ok, so seems problem was to do with using ExtremeManagement for SCP / SFTP as local FTP worked fine.

Issue I have now is that I wanted to add an IP address to each of the Rbridges. There is currently 8, and when I try and create interface ve 1999 on another Rbridge I get the following error:

code:
Col-xxx-VSP-Sw11(config-rbridge-id-12)# interface Ve 1999
Generic NSM Backend Error


Any idea what might be causing that?

Thanks
Userlevel 2
As it states, the message is Generic, however what I suspect could be happening is that the VLAN was not correctly provisioned on this RB. You can verify this by issuing

code:
show vlan brief


If this is the case you can try deleting/recreating the VLAN or reloading the box to see if the error persists.
Userlevel 5
Hi Michael,

The output of the command you requested is below.

My understanding (rightly or wrongly) is that I'm running the VDX in logical chassis mode, so from a layer 2 perspective creating the VLAN 1999 should exist on every switch.

When I going to 'Rbridge 12' it effectively puts me into the layer 3 router configuration, to which I want to create an IP address on each Rbridge for that VLAN, and where I'm hitting the error.

If on Rbridge 12 I pick a VLAN, say 151, that doesn't have an L3 address configured anywhere this works without a problem. Seems to be just related to ve 1999 that I have an IP address configured on Rbridge 11.

I'm just in the process of upgrading the switches via USB, as without being able to configure the IP's on the other Rbridge's I couldn't do it over the network.

Going from 6.02 to 7.0.2b, and then to 7.2.

The action of upgrading will reboot the switches, so will try again after that and report back.

code:
Col-xx-VSP-Sw11# show vlan brief
Total Number of VLANs configured : 23
Total Number of VLANs provisioned : 23
Total Number of VLANs unprovisioned : 0
VLAN Name State Ports Classification
(F)-FCoE (u)-Untagged
(R)-RSPAN (c)-Converged
(T)-TRANSPARENT (t)-Tagged
================ =============== ========================== =============== ====================
1 default ACTIVE Po 10(t)
Po 11(t)
Po 64(t)
22 VLAN0022 INACTIVE(member port down) Po 11(t)
30 VLAN0030 INACTIVE(member port down) Po 11(t)
64 VLAN0064 INACTIVE(member port down) Po 11(t)
70 VLAN0070 INACTIVE(member port down) Po 11(t)
71 VLAN0071 INACTIVE(member port down) Po 11(t)
102 VLAN0102 INACTIVE(member port down) Po 11(t)
146 VLAN0146 INACTIVE(member port down) Po 11(t)
147 VLAN0147 INACTIVE(member port down) Po 11(t)
148 VLAN0148 INACTIVE(member port down) Po 11(t)
149 VLAN0149 INACTIVE(member port down) Po 11(t)
150 VLAN0150 INACTIVE(member port down) Po 11(t)
151 VLAN0151 INACTIVE(member port down) Po 11(t)
199 VLAN0199 INACTIVE(member port down) Po 11(t)
240 VLAN0240 INACTIVE(member port down) Po 11(t)
252 VLAN0252 INACTIVE(member port down) Po 11(t)
1002(F) VLAN1002 INACTIVE(no member port)
1164 VLAN1164 ACTIVE Po 10(t)
Po 64(t)
1264 VLAN1264 ACTIVE Po 10(t)
Po 64(t)
1999 VLAN1999 ACTIVE Po 10(t)
2002 VLAN2002 INACTIVE(member port down) Po 11(t)
2003 VLAN2003 INACTIVE(member port down) Po 11(t)
3333 VLAN3333 INACTIVE(member port down) Po 11(t)


Many thanks,

Martin
Userlevel 2
Martin,

Your understanding is correct. VLANs are created globally and should be active on all RBs in a VCS fabric. My point is that the NSM backend error is likely caused by the RB not being in sync with this particular VLAN. In order to force this sync, you can remove and re-add the VLAN or reload the offending device so that it performs a config replay.
Userlevel 5
Hi Michael,

Your thoughts are correct. The reboot, and / or action of upgrading the switches seems to have corrected the problem and subsequently been able to create an interface ve 1999 on all the remaining Rbridges,

Thanks to you both for your perseverance.
Userlevel 2
Thanks Martin,

Glad to hear that resolved your issue.

Michael Morey
Principal Technical Support Engineer

Reply