Switches lose connection to ExtremeControl 7.0 .4.29 (TAFKAN) while performing DB backup

[MartinS1]

Hi there,

last week we decided to upgrade to NetSight Version 7.0.4.29. 😄

Since then, every time the NetSight DB is backuped, all switches appear as "Device Down" for a short period of time. But long enough for the system to send out the e-mail alarms.
As we've got 900+ Switches descovered, there are a lot of e-mails sent 😕

Does anyone have the same problem? Or should I open a GTAC case?!

This is the NetSight Error Log during the Upgrade:
2016-08-23 22:49:39,509 ERROR [com.extreme.common.poller.PollerUtils] processThrowable [3 updateDeviceStatus - Error sending device change messages] details:
com.enterasys.netsight.api.lockmgr.NSLockException: Could not acquire lock.
at com.enterasys.netsight.api.lockmgr.NSLock.acquire(NSLock.java:166)
at com.enterasys.netsight.server.ServerContext.acquireDeviceTreeWriteLock(ServerContext.java:750)
at com.enterasys.netsight.server.device.DeviceServiceImpl.updateObjectsInternal(DeviceServiceImpl.java:346)
at com.enterasys.netsight.server.device.DeviceServiceImpl.updateDeviceStatus(DeviceServiceImpl.java:318)
at com.extreme.common.poller.PollerUtils.updateDeviceStatus(PollerUtils.java:216)
at com.extreme.common.poller.SnmpStatusPollerTask.complete(SnmpStatusPollerTask.java:194)
at com.extreme.common.network.task.Task.complete(Task.java:553)
at com.extreme.common.network.request.RequestSnmp.processRequest(RequestSnmp.java:110)
at com.extreme.common.snmp.SnmpEngine$SnmpWrapper.doComplete(SnmpEngine.java:272)
at com.extreme.common.snmp.SnmpEngine$SnmpWrapper.access$300(SnmpEngine.java:156)
at com.extreme.common.snmp.SnmpEngine$SnmpWrapper$1.run(SnmpEngine.java:195)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2016-08-23 22:49:40,119 ERROR [com.extreme.common.poller.PollerUtils] processThrowable [2 updateDeviceStatus - Error sending device change messages] Message:Could not acquire lock. Cause:null

Called from:
Line:194 com.extreme.common.poller.SnmpStatusPollerTask method:complete
Line:553 com.extreme.common.network.task.Task method:complete
Line:110 com.extreme.common.network.request.RequestSnmp method:processRequest
Line:272 com.extreme.common.snmp.SnmpEngine$SnmpWrapper method:doComplete
Line:156 com.extreme.common.snmp.SnmpEngine$SnmpWrapper method:access$300
Line:195 com.extreme.common.snmp.SnmpEngine$SnmpWrapper$1 method:run
2016-08-23 22:49:40,119 ERROR [com.extreme.common.poller.PollerUtils] processThrowable [3 updateDeviceStatus - Error sending device change messages] Message:Could not acquire lock. Cause:null

Called from:
Line:194 com.extreme.common.poller.SnmpStatusPollerTask method:complete
Line:553 com.extreme.common.network.task.Task method:complete
Line:110 com.extreme.common.network.request.RequestSnmp method:processRequest
Line:272 com.extreme.common.snmp.SnmpEngine$SnmpWrapper method:doComplete
Line:156 com.extreme.common.snmp.SnmpEngine$SnmpWrapper method:access$300
Line:195 com.extreme.common.snmp.SnmpEngine$SnmpWrapper$1 method:run
2016-08-23 22:49:40,119 ERROR [com.extreme.common.poller.PollerUtils] processThrowable [2 updateDeviceStatus - Error sending device change messages] Message:Could not acquire lock. Cause:null

Called from:
Line:194 com.extreme.common.poller.SnmpStatusPollerTask method:complete
Line:553 com.extreme.common.network.task.Task method:complete
Line:110 com.extreme.common.network.request.RequestSnmp method:processRequest
Line:272 com.extreme.common.snmp.SnmpEngine$SnmpWrapper method:doComplete
Line:156 com.extreme.common.snmp.SnmpEngine$SnmpWrapper method:access$300
Line:195 com.extreme.common.snmp.SnmpEngine$SnmpWrapper$1 method:run
2016-08-23 22:49:40,119 ERROR [com.extreme.common.poller.PollerUtils] processThrowable [2 updateDeviceStatus - Error sending device change messages] Message:Could not acquire lock. Cause:null

Called from:
Line:194 com.extreme.common.poller.SnmpStatusPollerTask method:complete
Line:553 com.extreme.common.network.task.Task method:complete
Line:110 com.extreme.common.network.request.RequestSnmp method:processRequest
Line:272 com.extreme.common.snmp.SnmpEngine$SnmpWrapper method:doComplete
Line:156 com.extreme.common.snmp.SnmpEngine$SnmpWrapper method:access$300
Line:195 com.extreme.common.snmp.SnmpEngine$SnmpWrapper$1 method:run
2016-08-23 22:49:39,820 ERROR [org.jboss.mq.server.JMSDestinationManager] The connection to client ID:98 failed.
2016-08-23 22:49:39,509 ERROR [com.extreme.common.poller.PollerUtils] processThrowable [5 updateDeviceStatus - Error sending device change messages] details:
com.enterasys.netsight.api.lockmgr.NSLockException: Could not acquire lock.
at com.enterasys.netsight.api.lockmgr.NSLock.acquire(NSLock.java:166)
at com.enterasys.netsight.server.ServerContext.acquireDeviceTreeWriteLock(ServerContext.java:750)
at com.enterasys.netsight.server.device.DeviceServiceImpl.updateObjectsInternal(DeviceServiceImpl.java:346)
at com.enterasys.netsight.server.device.DeviceServiceImpl.updateDeviceStatus(DeviceServiceImpl.java:318)
at com.extreme.common.poller.PollerUtils.updateDeviceStatus(PollerUtils.java:216)
at com.extreme.common.poller.SnmpStatusPollerTask.complete(SnmpStatusPollerTask.java:194)
at com.extreme.common.network.task.Task.complete(Task.java:553)
at com.extreme.common.network.request.RequestSnmp.processRequest(RequestSnmp.java:110)
at com.extreme.common.snmp.SnmpEngine$SnmpWrapper.doComplete(SnmpEngine.java:272)
at com.extreme.common.snmp.SnmpEngine$SnmpWrapper.access$300(SnmpEngine.java:156)
at com.extreme.common.snmp.SnmpEngine$SnmpWrapper$1.run(SnmpEngine.java:195)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2016-08-23 22:49:39,509 ERROR [com.extreme.common.poller.PollerUtils] processThrowable [2 updateDeviceStatus - Error sending device change messages] details:
com.enterasys.netsight.api.lockmgr.NSLockException: Could not acquire lock.
at com.enterasys.netsight.api.lockmgr.NSLock.acquire(NSLock.java:166)
at com.enterasys.netsight.server.ServerContext.acquireDeviceTreeWriteLock(ServerContext.java:750)
at com.enterasys.netsight.server.device.DeviceServiceImpl.updateObjectsInternal(DeviceServiceImpl.java:346)
at com.enterasys.netsight.server.device.DeviceServiceImpl.updateDeviceStatus(DeviceServiceImpl.java:318)
at com.extreme.common.poller.PollerUtils.updateDeviceStatus(PollerUtils.java:216)
at com.extreme.common.poller.SnmpStatusPollerTask.complete(SnmpStatusPollerTask.java:194)
at com.extreme.common.network.task.Task.complete(Task.java:553)
at com.extreme.common.network.request.RequestSnmp.processRequest(RequestSnmp.java:110)
at com.extreme.common.snmp.SnmpEngine$SnmpWrapper.doComplete(SnmpEngine.java:272)
at com.extreme.common.snmp.SnmpEngine$SnmpWrapper.access$300(SnmpEngine.java:156)
at com.extreme.common.snmp.SnmpEngine$SnmpWrapper$1.run(SnmpEngine.java:195)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2016-08-23 22:49:39,509 ERROR [com.extreme.common.poller.PollerUtils] processThrowable [2 updateDeviceStatus - Error sending device change messages] details:
com.enterasys.netsight.api.lockmgr.NSLockException: Could not acquire lock.
at com.enterasys.netsight.api.lockmgr.NSLock.acquire(NSLock.java:166)
at com.enterasys.netsight.server.ServerContext.acquireDeviceTreeWriteLock(ServerContext.java:750)
at com.enterasys.netsight.server.device.DeviceServiceImpl.updateObjectsInternal(DeviceServiceImpl.java:346)
at com.enterasys.netsight.server.device.DeviceServiceImpl.updateDeviceStatus(DeviceServiceImpl.java:318)
at com.extreme.common.poller.PollerUtils.updateDeviceStatus(PollerUtils.java:216)
at com.extreme.common.poller.SnmpStatusPollerTask.complete(SnmpStatusPollerTask.java:194)
at com.extreme.common.network.task.Task.complete(Task.java:553)
at com.extreme.common.network.request.RequestSnmp.processRequest(RequestSnmp.java:110)
at com.extreme.common.snmp.SnmpEngine$SnmpWrapper.doComplete(SnmpEngine.java:272)
at com.extreme.common.snmp.SnmpEngine$SnmpWrapper.access$300(SnmpEngine.java:156)
at com.extreme.common.snmp.SnmpEngine$SnmpWrapper$1.run(SnmpEngine.java:195)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2016-08-23 22:49:39,509 ERROR [com.extreme.common.poller.PollerUtils] processThrowable [4 updateDeviceStatus - Error sending device change messages] details:
com.enterasys.netsight.api.lockmgr.NSLockException: Could not acquire lock.
at com.enterasys.netsight.api.lockmgr.NSLock.acquire(NSLock.java:166)
at com.enterasys.netsight.server.ServerContext.acquireDeviceTreeWriteLock(ServerContext.java:750)
at com.enterasys.netsight.server.device.DeviceServiceImpl.updateObjectsInternal(DeviceServiceImpl.java:346)
at com.enterasys.netsight.server.device.DeviceServiceImpl.updateDeviceStatus(DeviceServiceImpl.java:318)
at com.extreme.common.poller.PollerUtils.updateDeviceStatus(PollerUtils.java:216)
at com.extreme.common.poller.SnmpStatusPollerTask.complete(SnmpStatusPollerTask.java:194)
at com.extreme.common.network.task.Task.complete(Task.java:553)
at com.extreme.common.network.request.RequestSnmp.processRequest(RequestSnmp.java:110)
at com.extreme.common.snmp.SnmpEngine$SnmpWrapper.doComplete(SnmpEngine.java:272)
at com.extreme.common.snmp.SnmpEngine$SnmpWrapper.access$300(SnmpEngine.java:156)
at com.extreme.common.snmp.SnmpEngine$SnmpWrapper$1.run(SnmpEngine.java:195)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

Thanks in advance!
Cheers,
Martin

[MartinS1]

Hi Guys,

GTAC provided us the guide for "Enterprise DB Update".

It works pretty well 🙂

Thanks!

[Tom_Currier]

The GTAC provided an Enterprise Backup solution for version 7.0.5.12 which should alleviate this problem.

[MartinS1]

Update:
Netsight is still loosing connections during DB Updates.
The Memory Usages is currently set to ~12GB, and the mySQL is using "my-huge.ini".

[Tom_Currier]

Martin;

I'm taking this case and will respond directly from there. As mentioned in prior threads, this appears to be a resource issue where we might be able to make some adjustments to memory allocation to solve this issue.

Tom