Switches lose connection to ExtremeControl 7.0 .4.29 (TAFKAN) while performing DB backup

  • 1
  • 1
  • Problem
  • Updated 2 years ago
  • Solved
Hi there,

last week we decided to upgrade to NetSight Version 7.0.4.29. :-D

Since then, every time the NetSight DB is backuped, all switches appear as "Device Down" for a short period of time. But long enough for the system to send out the e-mail alarms.
As we've got 900+ Switches descovered, there are a lot of e-mails sent :-/

Does anyone have the same problem? Or should I open a GTAC case?!

This is the NetSight Error Log during the Upgrade:
2016-08-23 22:49:39,509 ERROR [com.extreme.common.poller.PollerUtils] processThrowable [3 updateDeviceStatus - Error sending device change messages] details: 
com.enterasys.netsight.api.lockmgr.NSLockException: Could not acquire lock.
at com.enterasys.netsight.api.lockmgr.NSLock.acquire(NSLock.java:166)
at com.enterasys.netsight.server.ServerContext.acquireDeviceTreeWriteLock(ServerContext.java:750)
at com.enterasys.netsight.server.device.DeviceServiceImpl.updateObjectsInternal(DeviceServiceImpl.java:346)
at com.enterasys.netsight.server.device.DeviceServiceImpl.updateDeviceStatus(DeviceServiceImpl.java:318)
at com.extreme.common.poller.PollerUtils.updateDeviceStatus(PollerUtils.java:216)
at com.extreme.common.poller.SnmpStatusPollerTask.complete(SnmpStatusPollerTask.java:194)
at com.extreme.common.network.task.Task.complete(Task.java:553)
at com.extreme.common.network.request.RequestSnmp.processRequest(RequestSnmp.java:110)
at com.extreme.common.snmp.SnmpEngine$SnmpWrapper.doComplete(SnmpEngine.java:272)
at com.extreme.common.snmp.SnmpEngine$SnmpWrapper.access$300(SnmpEngine.java:156)
at com.extreme.common.snmp.SnmpEngine$SnmpWrapper$1.run(SnmpEngine.java:195)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2016-08-23 22:49:40,119 ERROR [com.extreme.common.poller.PollerUtils] processThrowable [2 updateDeviceStatus - Error sending device change messages] Message:Could not acquire lock. Cause:null

Called from:
Line:194 com.extreme.common.poller.SnmpStatusPollerTask method:complete
Line:553 com.extreme.common.network.task.Task method:complete
Line:110 com.extreme.common.network.request.RequestSnmp method:processRequest
Line:272 com.extreme.common.snmp.SnmpEngine$SnmpWrapper method:doComplete
Line:156 com.extreme.common.snmp.SnmpEngine$SnmpWrapper method:access$300
Line:195 com.extreme.common.snmp.SnmpEngine$SnmpWrapper$1 method:run
2016-08-23 22:49:40,119 ERROR [com.extreme.common.poller.PollerUtils] processThrowable [3 updateDeviceStatus - Error sending device change messages] Message:Could not acquire lock. Cause:null

Called from:
Line:194 com.extreme.common.poller.SnmpStatusPollerTask method:complete
Line:553 com.extreme.common.network.task.Task method:complete
Line:110 com.extreme.common.network.request.RequestSnmp method:processRequest
Line:272 com.extreme.common.snmp.SnmpEngine$SnmpWrapper method:doComplete
Line:156 com.extreme.common.snmp.SnmpEngine$SnmpWrapper method:access$300
Line:195 com.extreme.common.snmp.SnmpEngine$SnmpWrapper$1 method:run
2016-08-23 22:49:40,119 ERROR [com.extreme.common.poller.PollerUtils] processThrowable [2 updateDeviceStatus - Error sending device change messages] Message:Could not acquire lock. Cause:null

Called from:
Line:194 com.extreme.common.poller.SnmpStatusPollerTask method:complete
Line:553 com.extreme.common.network.task.Task method:complete
Line:110 com.extreme.common.network.request.RequestSnmp method:processRequest
Line:272 com.extreme.common.snmp.SnmpEngine$SnmpWrapper method:doComplete
Line:156 com.extreme.common.snmp.SnmpEngine$SnmpWrapper method:access$300
Line:195 com.extreme.common.snmp.SnmpEngine$SnmpWrapper$1 method:run
2016-08-23 22:49:40,119 ERROR [com.extreme.common.poller.PollerUtils] processThrowable [2 updateDeviceStatus - Error sending device change messages] Message:Could not acquire lock. Cause:null

Called from:
Line:194 com.extreme.common.poller.SnmpStatusPollerTask method:complete
Line:553 com.extreme.common.network.task.Task method:complete
Line:110 com.extreme.common.network.request.RequestSnmp method:processRequest
Line:272 com.extreme.common.snmp.SnmpEngine$SnmpWrapper method:doComplete
Line:156 com.extreme.common.snmp.SnmpEngine$SnmpWrapper method:access$300
Line:195 com.extreme.common.snmp.SnmpEngine$SnmpWrapper$1 method:run
2016-08-23 22:49:39,820 ERROR [org.jboss.mq.server.JMSDestinationManager] The connection to client ID:98 failed.
2016-08-23 22:49:39,509 ERROR [com.extreme.common.poller.PollerUtils] processThrowable [5 updateDeviceStatus - Error sending device change messages] details:
com.enterasys.netsight.api.lockmgr.NSLockException: Could not acquire lock.
at com.enterasys.netsight.api.lockmgr.NSLock.acquire(NSLock.java:166)
at com.enterasys.netsight.server.ServerContext.acquireDeviceTreeWriteLock(ServerContext.java:750)
at com.enterasys.netsight.server.device.DeviceServiceImpl.updateObjectsInternal(DeviceServiceImpl.java:346)
at com.enterasys.netsight.server.device.DeviceServiceImpl.updateDeviceStatus(DeviceServiceImpl.java:318)
at com.extreme.common.poller.PollerUtils.updateDeviceStatus(PollerUtils.java:216)
at com.extreme.common.poller.SnmpStatusPollerTask.complete(SnmpStatusPollerTask.java:194)
at com.extreme.common.network.task.Task.complete(Task.java:553)
at com.extreme.common.network.request.RequestSnmp.processRequest(RequestSnmp.java:110)
at com.extreme.common.snmp.SnmpEngine$SnmpWrapper.doComplete(SnmpEngine.java:272)
at com.extreme.common.snmp.SnmpEngine$SnmpWrapper.access$300(SnmpEngine.java:156)
at com.extreme.common.snmp.SnmpEngine$SnmpWrapper$1.run(SnmpEngine.java:195)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2016-08-23 22:49:39,509 ERROR [com.extreme.common.poller.PollerUtils] processThrowable [2 updateDeviceStatus - Error sending device change messages] details:
com.enterasys.netsight.api.lockmgr.NSLockException: Could not acquire lock.
at com.enterasys.netsight.api.lockmgr.NSLock.acquire(NSLock.java:166)
at com.enterasys.netsight.server.ServerContext.acquireDeviceTreeWriteLock(ServerContext.java:750)
at com.enterasys.netsight.server.device.DeviceServiceImpl.updateObjectsInternal(DeviceServiceImpl.java:346)
at com.enterasys.netsight.server.device.DeviceServiceImpl.updateDeviceStatus(DeviceServiceImpl.java:318)
at com.extreme.common.poller.PollerUtils.updateDeviceStatus(PollerUtils.java:216)
at com.extreme.common.poller.SnmpStatusPollerTask.complete(SnmpStatusPollerTask.java:194)
at com.extreme.common.network.task.Task.complete(Task.java:553)
at com.extreme.common.network.request.RequestSnmp.processRequest(RequestSnmp.java:110)
at com.extreme.common.snmp.SnmpEngine$SnmpWrapper.doComplete(SnmpEngine.java:272)
at com.extreme.common.snmp.SnmpEngine$SnmpWrapper.access$300(SnmpEngine.java:156)
at com.extreme.common.snmp.SnmpEngine$SnmpWrapper$1.run(SnmpEngine.java:195)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2016-08-23 22:49:39,509 ERROR [com.extreme.common.poller.PollerUtils] processThrowable [2 updateDeviceStatus - Error sending device change messages] details:
com.enterasys.netsight.api.lockmgr.NSLockException: Could not acquire lock.
at com.enterasys.netsight.api.lockmgr.NSLock.acquire(NSLock.java:166)
at com.enterasys.netsight.server.ServerContext.acquireDeviceTreeWriteLock(ServerContext.java:750)
at com.enterasys.netsight.server.device.DeviceServiceImpl.updateObjectsInternal(DeviceServiceImpl.java:346)
at com.enterasys.netsight.server.device.DeviceServiceImpl.updateDeviceStatus(DeviceServiceImpl.java:318)
at com.extreme.common.poller.PollerUtils.updateDeviceStatus(PollerUtils.java:216)
at com.extreme.common.poller.SnmpStatusPollerTask.complete(SnmpStatusPollerTask.java:194)
at com.extreme.common.network.task.Task.complete(Task.java:553)
at com.extreme.common.network.request.RequestSnmp.processRequest(RequestSnmp.java:110)
at com.extreme.common.snmp.SnmpEngine$SnmpWrapper.doComplete(SnmpEngine.java:272)
at com.extreme.common.snmp.SnmpEngine$SnmpWrapper.access$300(SnmpEngine.java:156)
at com.extreme.common.snmp.SnmpEngine$SnmpWrapper$1.run(SnmpEngine.java:195)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2016-08-23 22:49:39,509 ERROR [com.extreme.common.poller.PollerUtils] processThrowable [4 updateDeviceStatus - Error sending device change messages] details:
com.enterasys.netsight.api.lockmgr.NSLockException: Could not acquire lock.
at com.enterasys.netsight.api.lockmgr.NSLock.acquire(NSLock.java:166)
at com.enterasys.netsight.server.ServerContext.acquireDeviceTreeWriteLock(ServerContext.java:750)
at com.enterasys.netsight.server.device.DeviceServiceImpl.updateObjectsInternal(DeviceServiceImpl.java:346)
at com.enterasys.netsight.server.device.DeviceServiceImpl.updateDeviceStatus(DeviceServiceImpl.java:318)
at com.extreme.common.poller.PollerUtils.updateDeviceStatus(PollerUtils.java:216)
at com.extreme.common.poller.SnmpStatusPollerTask.complete(SnmpStatusPollerTask.java:194)
at com.extreme.common.network.task.Task.complete(Task.java:553)
at com.extreme.common.network.request.RequestSnmp.processRequest(RequestSnmp.java:110)
at com.extreme.common.snmp.SnmpEngine$SnmpWrapper.doComplete(SnmpEngine.java:272)
at com.extreme.common.snmp.SnmpEngine$SnmpWrapper.access$300(SnmpEngine.java:156)
at com.extreme.common.snmp.SnmpEngine$SnmpWrapper$1.run(SnmpEngine.java:195)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

Thanks in advance!
Cheers,
Martin
Photo of MartinS

MartinS

  • 430 Points 250 badge 2x thumb
  • frustrated

Posted 2 years ago

  • 1
  • 1
Photo of Jeremy

Jeremy, Embassador

  • 9,788 Points 5k badge 2x thumb
I am no expert, but it sounds like your server might be running out of resources... Do you monitor the CPU / disk utilization at all? 
Photo of Thomas, Frank

Thomas, Frank, Employee

  • 1,902 Points 1k badge 2x thumb
How big are your Database backups also how long do they take to complete? The completion time should be in console event logs.
Photo of MartinS

MartinS

  • 430 Points 250 badge 2x thumb
I'm going to check tomorrow, as it's the system of a customer I don't have access to it every day ;-)
(Edited)
Photo of MartinS

MartinS

  • 430 Points 250 badge 2x thumb
The 1st backup takes about 3 minutes (~125MB), the 2nd takes about 51 minutes (~17GB)


Just before the finish of the second backup, there is a massive utilization of the disks.
This seems to be the problem which was not there before the update.

I'm going to open a GTAC-case :-/

regards,
Martin
Photo of Tom Currier

Tom Currier, Employee

  • 720 Points 500 badge 2x thumb
Martin;

I'm taking this case and will respond directly from there.  As mentioned in prior threads, this appears to be a resource issue where we might be able to make some adjustments to memory allocation to solve this issue.

Tom
Photo of MartinS

MartinS

  • 430 Points 250 badge 2x thumb
Update:
Netsight is still loosing connections during DB Updates.
The Memory Usages is currently set to ~12GB, and the mySQL is using "my-huge.ini".
Photo of Tom Currier

Tom Currier, Employee

  • 720 Points 500 badge 2x thumb
The GTAC provided an Enterprise Backup solution for version 7.0.5.12 which should alleviate this problem.
Photo of MartinS

MartinS

  • 430 Points 250 badge 2x thumb
Hi Guys,

GTAC provided us the guide for "Enterprise DB Update".

It works pretty well :-)

Thanks!