topic Re: 3 switches won't back up config in xmc (File not found / missing) in ExtremeCloud IQ- Site Engine Management Center

3 switches won't back up config in xmc (File not found / missing)

Keith9 — Mon, 29 Mar 2021 19:59:56 GMT

I have daily config backup configured for 7 am for two groups, core and access. I have all our switches sucessfully backing except 3 of them. I have 11 switch stacks in our access switch group and 3 of these stacks have the this message

File Not Found/Missing -Error: Could not connect with TFTP server at 10.1.0.110

File Not Found/Missing -Error: Could not connect with TFTP server at 10.1.0.110

File Not Found/Missing - The device was unable to contact the TFTP server. Check that the TFTP server is runinng and connectivity is okay.

So the question poses, why just 3 switch stacks showing this error? I have an ACL on all fo them SSH-access.pol, and I put the netsight server IP in it (10.1.0.110/32) and verified it worked by opening a terminal session to every switch stack. This resolved my issue with the bulk of them, besides changing the SSH account after that broke when I changed to radius authentication.

So I have 2 valid core switch stacks backing up and 8 access switch stacks backing up. I’m not sure whats up with the last 3. They can ping and log in from the XMC web GUI. There’s nothing impeding their path to XMC.

Thanks for any helpful hints.

Re: 3 switches won't back up config in xmc (File not found / missing)

Stefan_K_ — Mon, 29 Mar 2021 21:00:50 GMT

I would check the inventory settings of these switches.

Is TFTP or SCP selected? Is the right script selected?

Re: 3 switches won't back up config in xmc (File not found / missing)

Keith9 — Mon, 29 Mar 2021 22:24:34 GMT

Every single switch is configured for TFTP (default), Script and ExtremeXOS - TFTP.

I can kick of an archive configuration command and in the switch logs I see the user login and save the config but it never makes it from the switch to the netsight server at 10.1.0.110.

I’m not sure what could be blocking it. I checked access lists on our core and found nothing specific for blocking these few switches to netsight. However the switches that don’t work can’t ping netsight.

In fact ONE of the switches when testing an snmp trap alert just would not inform netsight. I had to change its TVInform tag to come from its loopback IP instead of its primary default vlan IP, and re-add it into netsight using our loopback ip scheme (192.168.255.x). That fixed snmp trap and alerting on the one switch but otherwise either TFTP or SCP, I cannot get the configuration archived.

Netsight server is a ubuntu vm. Do you know anywhere in ubuntu where there could be a firewall or access list? I supposed somewhere in /etc right? I’m just trying to figure out if I put anything on there to block something that may have come up on a vulnerability scanner and then inadvertently blocked 3 legitimate switches. Its certainly not on our core.

Were on Extreme Management Center 8.5.2.6.

Re: 3 switches won't back up config in xmc (File not found / missing)

Stefan_K_ — Mon, 29 Mar 2021 22:28:33 GMT

The affected switches can’t ping Netsight but all other switches can? This is some good symptom. Are there multiple VRs on the switch? Can you ping the core-router?

Re: 3 switches won't back up config in xmc (File not found / missing)

Keith9 — Mon, 29 Mar 2021 22:34:19 GMT

Yes they can ping the core, they can ping our domain controllers and other servers in the same network (10.1.x.x) but they can’t ping 10.1.0.110).

A traceroute to 10.1.0.110 from an affected switch shows it goes to our core and then stops there. The vm servers are plugged into our core so the second hop should be the netsight box. If I check with a switch that can ping, it does go to the core and yes the second hop is the netsight box.

I’m not sure where in the netsight ubuntu virtual machine to look. It doesn’t seem like its in /etc/ufw

Years ago I seem to remember a vulnerability in netsight and I did something to block our scanners from accessing it, but I forget where. It doesn’t seem like its apache so its not an apache or .htaccess config, so I’m not sure if thats related or not.

Re: 3 switches won't back up config in xmc (File not found / missing)

Keith9 — Mon, 29 Mar 2021 22:38:42 GMT

Let me ask you this, if I want to remove an acl1 from a port to test, how…

configure access-list acl1 ports 1:13 ingress

How can I take acl1 off port 1:13 in the above example .

Re: 3 switches won't back up config in xmc (File not found / missing)

Keith9 — Mon, 29 Mar 2021 22:56:19 GMT

Nevermind, a lot of trial and error with tab completion , found it

unconfigure access-list ports 1:13 ingress

anyway that didn’t make a difference.

When logged into netsight server at 10.1.0.110 it can ping and traceroute fine to a troubled switch. The switch cannot ping or traceroute to netsight. It can ping from its loopback0 ip if I use the ping command with the from.

One VR-Default. VR-Mgmt exists on the switch too, but isn’t used for anything.

Odd thing is I can have both sides ping at the same time, you’d think they’d see each others packets hitting at the same time and say Oh ok good… but no only get a response from netsight pinging out using the ping command at the ubuntu cli.

Re: 3 switches won't back up config in xmc (File not found / missing)

Stefan_K_ — Mon, 29 Mar 2021 23:01:43 GMT

Try

sudo ufw status

to check the current rules or

sudo ufw disable

to simply disable it. Might also be something in iptables?

unconfigure access-list acl1 ports 1:13 ingress

Re: 3 switches won't back up config in xmc (File not found / missing)

Keith9 — Mon, 29 Mar 2021 23:06:06 GMT

sudo ufw staus

stats: inactive

iptables --list

I dont know, its completely bizarre.

Re: 3 switches won't back up config in xmc (File not found / missing)

Stefan_K_ — Mon, 29 Mar 2021 23:10:03 GMT

Hmm, interesting. Maybe it’s time for some tcpdumps and port-mirros.. 🙂

Re: 3 switches won't back up config in xmc (File not found / missing)

Keith9 — Mon, 29 Mar 2021 23:23:05 GMT

Ok so from a trouble switch when I just do a plain ping 10.1.0.110 its trying to source it from 192.168.100.50 which is the ip of the switch on our transport vlan.

If I source it from 10.50.0.100 (vlan 1 / Default) or 192.168.255.50 (loopback0) or from our vlan that has an ospf adjancency with a cisco router with LTE tunnel backup to HQ (10.200.1.17), all the pings succeed.

For some reason, by default the switch wants to use the vlan WAN IP of our 40/40mbps metro e transport IP and that does not return.

I’m not sure where in its brain it wants to source from that IP from default. I thought it would start at vlan 1 (the top of the list). Not vlan 296 (our wan tranport out).

Re: 3 switches won't back up config in xmc (File not found / missing)

StephanH — Mon, 29 Mar 2021 23:31:48 GMT

Hello Keith,

can you show your routing table to us, please?

Re: 3 switches won't back up config in xmc (File not found / missing)

Stefan_K_ — Mon, 29 Mar 2021 23:32:00 GMT

Aaah, Transfer-net? Didn’t expect that between Edge and Core Switches. The switch uses the ipaddress of the egress interface, so it uses 192.168.100.50 in this case. I had the exact same case many months ago.

How should the switch know which IP-Address to use… 🙂 On EOS you could define an mgmt IP-Address, but on EXOS that’s not possible.

You have to possibilities imo:

configure a switch mgmt VLAN and create a new VR for this. Add the switch with this ip address to XMC and create a new TFTP script that uses the newly created VR. Might even work with the exisiting VR-Mgmt.
Make the transfer-net reachable from XMC

Re: 3 switches won't back up config in xmc (File not found / missing)

Keith9 — Mon, 29 Mar 2021 23:49:29 GMT

Ok well talking this through in the thread here got me looking more closely at the routing tables. Netsight VM traffic was exiting our 2nd core switch (all vm’s are mlag’d between 2 cores). Some VM’s go out one core, some the other.

The second core had that transport network advertised in its OSFP configuration and an IP interface on it, however that WAN is not connected. We were going to earmark a port on it to go “swing the cable” in the event something happened to our primary core switch, however we don’t have any ports for that right now and we have a different, alternative backup path should we need to reach those locations out core 2.

So the 192.168.100.x/24 transport network that 3 remote sites were on, were not making it through to the second core switch. I mean you could ping and trace PAST that network, but the EXOS switch INSISTED (as Stefan K. shows some examples) to use the interface in that 192.168.100.x network. So to end users and servers, traditionally they noticed no issue. They live and work in different IP networks that are all in the routing table.

So I removed that VLAN from core 2’s OSPF configuration and now it traverses an OSPF link we have between both cores and out that WAN provider that is physically connected into our core 1 switch stack.

(Our core switch stacks are X690’s, our access stacks are X450G2’s or 5520’s depending on location).

So in actuality this was a routing issue and now I was able to retrieve configurations successfully.

Thanks for talking me through it. Sometimes you don’t notice the mundane details until you start typing it out and having a casual conversation.