topic RE: X690 stack stopped responding to SNMP earlier in the month in ExtremeSwitching (EXOS/Switch Engine)

X690 stack stopped responding to SNMP earlier in the month

Keith9 — Thu, 14 Oct 2021 15:55:00 GMT

Went to research some bandwidth utilization on a x690 core switch stack in cacti and notice the graphs are all blank. Upon further diagnosis this seemed to occur between 9/30 and 10/3 with graphs slowly loosing resolution. Verified with the diagnostics in the app that there is no longer an SNMP response, something that's been working fine since it's deployment in 2018. Also tried an snmp tool from my workstation and verified in XMC that the device is logged for loss of SNMP contact around 3 am on 10/3.

I verified with these tools that all other extreme switches SNMP is working fine, including our second core stack of X690s. We have those plus 450G2's and 5520's, all working fine with our extreme snmpv3 credentials.

We opened a case and so far they just had us reset counters and sh management, but it's been dead silence since. Anyone ever see this happen?

RE: X690 stack stopped responding to SNMP earlier in the month

Keith_Obermeier — Thu, 14 Oct 2021 16:38:00 GMT

Did you do an XOS update on the 690 around the time it stopped responding?

Recent XOS releases for the 690 have changed it's snmp identity and associated MIBs. you may need to remove/readd it with a fresh snmpwalk or updated MIBs.

In my case XMC will no longer treat a 690 as policy capable on my current firmware. I am planning on updating it during Thanksgiving weekend to try to fix it.

RE: X690 stack stopped responding to SNMP earlier in the month

Keith9 — Fri, 15 Oct 2021 03:18:00 GMT

Nope, no updates on a few years actually. I think it has somewhere like 1000+ day uptime.

RE: X690 stack stopped responding to SNMP earlier in the month

Keith9 — Tue, 19 Oct 2021 17:19:00 GMT

Resolved with the following..

#restart process snmpMaster
Step 1: terminating process snmpMaster gracefully ...
Step 2: starting process snmpMaster ...
Restarted process snmpMaster successfully

Now both cacti graphing and netsight (XMC) can see the switch again. Also the command sh management displays in its entirety, rather than displaying 2/3rds hanging for 10 seconds and then displaying the rest.

Weird issue, but I'm thankful for the modularity of the linux core and the ability to restart a process without taking an entire switch down.