ā09-23-2019 03:51 PM
SR2348 on the latest firmware. All switches are in battery backups. Cannot be a power issue as two switches are plugged into the same power strip, one reboots and the other does not.
ā10-20-2019 05:20 PM
For anyone watching this there is a suspected memory leak for the configuration that I am running on my switches. Over the last few weeks the memory has steady climbed and is now at 95% on a group of switches. We are running console logging on the switches hoping to get some insight as the syslog and persistent logs have no data. If that fails then I guess I will have to setup a job to reboot the switches during off hours (not good) to prevent them from crashing during the day.
ā09-26-2019 03:08 PM
Sam,
Tell Kyle I said hi. Have worked with him in several cases. I have a case open for a few weeks now. It just got escalated to Ryan Mundell, Kyle's neighbor.
My main reason for posting here was to see if others in the community were having this issue or was it unique to my setup. We still don't know if others are having an issue but we believe we have discovered the case. In every case of a switch reboot (10 so far) all have had the memory exceed 95% just prior to the reboot. So at this point it appears to be a memory leak. There was a known issue in version 1.0.1.25 but I am on 1.0.1.26. This may be a case where RSTP or another service I am running is causing it as to why others are not having the issue.
Unfortunately because of the how the switches are rebooting there is no data in the logs. We are currently looking into setting up console loggers to try and capture the data up to the last moment.
Thanks for discussing this with Kyle, didn't mean to waste your time like that. In the future I will make sure to note there is a case open.
Doug
ā09-26-2019 01:59 PM
I reviewed this case with Kyle Fahrenbach, one of our expert technicians, and he asked that you open a case for this issue and send us tech data from that switch. I can't speak to other customer cases, but this is the first we've heard of this issue in general.
ā09-24-2019 05:33 PM
It is set to logging persistent 7.
#show logging
Logging Client Local Port : 514
Logging Client Source Interface : (not configured)
CLI Command Logging : disabled
Console Logging : enabled
Console Logging Severity Filter : debug
Buffered Logging : enabled
Buffered Logging Severity Filter : debug
Persistent Logging : enabled
Persistent Logging Severity Filter : debug
Syslog Logging : enabled
Log Messages Received : 178418
When I run the "show logging persistent" command it only has logs since the device rebooted.
The issue is that the logging is broken. After the reboot there are no current logs prior to the reboot. There are logs from months ago in some switches and nothing in others. If I had persistent set in the past those logs are still there but nothing is being saved prior to the reboot. I am currently dealing with a triple issue, no logs, spanning tree storm on bootup and random reboots of switches for some unknown reason.
And nothing makes it to the syslog server to indicate why the switch reboots. I am seeing logs there but they just stop and then start back a few minutes later with no entries that indicate why.
We have several of our switches in a loop: CORE 1 <> SWITCH 1 <> SWITCH 2 <> SWITCH 3 <> CORE 2
These are not stacked. If switch 2 reboots it causes a brief storm since the uplink ports forward traffic during bootup prior to the OS loading. I have tickets open for all three issues but all are stuck with no progress.
I find it hard to believe that I am the only customer with this issue.
Am I the only one using these in an enterprise environment?
That is why I reached out here to see if others are having this issue.
I am in the process of preparing to return all 102 of my switches since TAC doesn't seem to care about fixing the issue. Random reboots in the middle of the day knocking off hundreds of users doesn't seem to be an important enough issue to worry about.