Switches lose connection to ExtremeControl 7.0 .4.29 (TAFKAN) while performing DB backup

[MartinS1]

Hi there,

last week we decided to upgrade to NetSight Version 7.0.4.29. 😄

Since then, every time the NetSight DB is backuped, all switches appear as "Device Down" for a short period of time. But long enough for the system to send out the e-mail alarms.
As we've got 900+ Switches descovered, there are a lot of e-mails sent 😕

Does anyone have the same problem? Or should I open a GTAC case?!

This is the NetSight Error Log during the Upgrade:
2016-08-23 22:49:39,509 ERROR [com.extreme.common.poller.PollerUtils] processThrowable [3 updateDeviceStatus - Error sending device change messages] details:
com.enterasys.netsight.api.lockmgr.NSLockException: Could not acquire lock.
at com.enterasys.netsight.api.lockmgr.NSLock.acquire(NSLock.java:166)
at com.enterasys.netsight.server.ServerContext.acquireDeviceTreeWriteLock(ServerContext.java:750)
at com.enterasys.netsight.server.device.DeviceServiceImpl.updateObjectsInternal(DeviceServiceImpl.java:346)
at com.enterasys.netsight.server.device.DeviceServiceImpl.updateDeviceStatus(DeviceServiceImpl.java:318)
at com.extreme.common.poller.PollerUtils.updateDeviceStatus(PollerUtils.java:216)
at com.extreme.common.poller.SnmpStatusPollerTask.complete(SnmpStatusPollerTask.java:194)
at com.extreme.common.network.task.Task.complete(Task.java:553)
at com.extreme.common.network.request.RequestSnmp.processRequest(RequestSnmp.java:110)
at com.extreme.common.snmp.SnmpEngine$SnmpWrapper.doComplete(SnmpEngine.java:272)
at com.extreme.common.snmp.SnmpEngine$SnmpWrapper.access$300(SnmpEngine.java:156)
at com.extreme.common.snmp.SnmpEngine$SnmpWrapper$1.run(SnmpEngine.java:195)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2016-08-23 22:49:40,119 ERROR [com.extreme.common.poller.PollerUtils] processThrowable [2 updateDeviceStatus - Error sending device change messages] Message:Could not acquire lock. Cause:null

Called from:
Line:194 com.extreme.common.poller.SnmpStatusPollerTask method:complete
Line:553 com.extreme.common.network.task.Task method:complete
Line:110 com.extreme.common.network.request.RequestSnmp method:processRequest
Line:272 com.extreme.common.snmp.SnmpEngine$SnmpWrapper method:doComplete
Line:156 com.extreme.common.snmp.SnmpEngine$SnmpWrapper method:access$300
Line:195 com.extreme.common.snmp.SnmpEngine$SnmpWrapper$1 method:run
2016-08-23 22:49:40,119 ERROR [com.extreme.common.poller.PollerUtils] processThrowable [3 updateDeviceStatus - Error sending device change messages] Message:Could not acquire lock. Cause:null

Called from:
Line:194 com.extreme.common.poller.SnmpStatusPollerTask method:complete
Line:553 com.extreme.common.network.task.Task method:complete
Line:110 com.extreme.common.network.request.RequestSnmp method:processRequest
Line:272 com.extreme.common.snmp.SnmpEngine$SnmpWrapper method:doComplete
Line:156 com.extreme.common.snmp.SnmpEngine$SnmpWrapper method:access$300
Line:195 com.extreme.common.snmp.SnmpEngine$SnmpWrapper$1 method:run
2016-08-23 22:49:40,119 ERROR [com.extreme.common.poller.PollerUtils] processThrowable [2 updateDeviceStatus - Error sending device change messages] Message:Could not acquire lock. Cause:null

Called from:
Line:194 com.extreme.common.poller.SnmpStatusPollerTask method:complete
Line:553 com.extreme.common.network.task.Task method:complete
Line:110 com.extreme.common.network.request.RequestSnmp method:processRequest
Line:272 com.extreme.common.snmp.SnmpEngine$SnmpWrapper method:doComplete
Line:156 com.extreme.common.snmp.SnmpEngine$SnmpWrapper method:access$300
Line:195 com.extreme.common.snmp.SnmpEngine$SnmpWrapper$1 method:run
2016-08-23 22:49:40,119 ERROR [com.extreme.common.poller.PollerUtils] processThrowable [2 updateDeviceStatus - Error sending device change messages] Message:Could not acquire lock. Cause:null

Called from:
Line:194 com.extreme.common.poller.SnmpStatusPollerTask method:complete
Line:553 com.extreme.common.network.task.Task method:complete
Line:110 com.extreme.common.network.request.RequestSnmp method:processRequest
Line:272 com.extreme.common.snmp.SnmpEngine$SnmpWrapper method:doComplete
Line:156 com.extreme.common.snmp.SnmpEngine$SnmpWrapper method:access$300
Line:195 com.extreme.common.snmp.SnmpEngine$SnmpWrapper$1 method:run
2016-08-23 22:49:39,820 ERROR [org.jboss.mq.server.JMSDestinationManager] The connection to client ID:98 failed.
2016-08-23 22:49:39,509 ERROR [com.extreme.common.poller.PollerUtils] processThrowable [5 updateDeviceStatus - Error sending device change messages] details:
com.enterasys.netsight.api.lockmgr.NSLockException: Could not acquire lock.
at com.enterasys.netsight.api.lockmgr.NSLock.acquire(NSLock.java:166)
at com.enterasys.netsight.server.ServerContext.acquireDeviceTreeWriteLock(ServerContext.java:750)
at com.enterasys.netsight.server.device.DeviceServiceImpl.updateObjectsInternal(DeviceServiceImpl.java:346)
at com.enterasys.netsight.server.device.DeviceServiceImpl.updateDeviceStatus(DeviceServiceImpl.java:318)
at com.extreme.common.poller.PollerUtils.updateDeviceStatus(PollerUtils.java:216)
at com.extreme.common.poller.SnmpStatusPollerTask.complete(SnmpStatusPollerTask.java:194)
at com.extreme.common.network.task.Task.complete(Task.java:553)
at com.extreme.common.network.request.RequestSnmp.processRequest(RequestSnmp.java:110)
at com.extreme.common.snmp.SnmpEngine$SnmpWrapper.doComplete(SnmpEngine.java:272)
at com.extreme.common.snmp.SnmpEngine$SnmpWrapper.access$300(SnmpEngine.java:156)
at com.extreme.common.snmp.SnmpEngine$SnmpWrapper$1.run(SnmpEngine.java:195)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2016-08-23 22:49:39,509 ERROR [com.extreme.common.poller.PollerUtils] processThrowable [2 updateDeviceStatus - Error sending device change messages] details:
com.enterasys.netsight.api.lockmgr.NSLockException: Could not acquire lock.
at com.enterasys.netsight.api.lockmgr.NSLock.acquire(NSLock.java:166)
at com.enterasys.netsight.server.ServerContext.acquireDeviceTreeWriteLock(ServerContext.java:750)
at com.enterasys.netsight.server.device.DeviceServiceImpl.updateObjectsInternal(DeviceServiceImpl.java:346)
at com.enterasys.netsight.server.device.DeviceServiceImpl.updateDeviceStatus(DeviceServiceImpl.java:318)
at com.extreme.common.poller.PollerUtils.updateDeviceStatus(PollerUtils.java:216)
at com.extreme.common.poller.SnmpStatusPollerTask.complete(SnmpStatusPollerTask.java:194)
at com.extreme.common.network.task.Task.complete(Task.java:553)
at com.extreme.common.network.request.RequestSnmp.processRequest(RequestSnmp.java:110)
at com.extreme.common.snmp.SnmpEngine$SnmpWrapper.doComplete(SnmpEngine.java:272)
at com.extreme.common.snmp.SnmpEngine$SnmpWrapper.access$300(SnmpEngine.java:156)
at com.extreme.common.snmp.SnmpEngine$SnmpWrapper$1.run(SnmpEngine.java:195)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2016-08-23 22:49:39,509 ERROR [com.extreme.common.poller.PollerUtils] processThrowable [2 updateDeviceStatus - Error sending device change messages] details:
com.enterasys.netsight.api.lockmgr.NSLockException: Could not acquire lock.
at com.enterasys.netsight.api.lockmgr.NSLock.acquire(NSLock.java:166)
at com.enterasys.netsight.server.ServerContext.acquireDeviceTreeWriteLock(ServerContext.java:750)
at com.enterasys.netsight.server.device.DeviceServiceImpl.updateObjectsInternal(DeviceServiceImpl.java:346)
at com.enterasys.netsight.server.device.DeviceServiceImpl.updateDeviceStatus(DeviceServiceImpl.java:318)
at com.extreme.common.poller.PollerUtils.updateDeviceStatus(PollerUtils.java:216)
at com.extreme.common.poller.SnmpStatusPollerTask.complete(SnmpStatusPollerTask.java:194)
at com.extreme.common.network.task.Task.complete(Task.java:553)
at com.extreme.common.network.request.RequestSnmp.processRequest(RequestSnmp.java:110)
at com.extreme.common.snmp.SnmpEngine$SnmpWrapper.doComplete(SnmpEngine.java:272)
at com.extreme.common.snmp.SnmpEngine$SnmpWrapper.access$300(SnmpEngine.java:156)
at com.extreme.common.snmp.SnmpEngine$SnmpWrapper$1.run(SnmpEngine.java:195)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2016-08-23 22:49:39,509 ERROR [com.extreme.common.poller.PollerUtils] processThrowable [4 updateDeviceStatus - Error sending device change messages] details:
com.enterasys.netsight.api.lockmgr.NSLockException: Could not acquire lock.
at com.enterasys.netsight.api.lockmgr.NSLock.acquire(NSLock.java:166)
at com.enterasys.netsight.server.ServerContext.acquireDeviceTreeWriteLock(ServerContext.java:750)
at com.enterasys.netsight.server.device.DeviceServiceImpl.updateObjectsInternal(DeviceServiceImpl.java:346)
at com.enterasys.netsight.server.device.DeviceServiceImpl.updateDeviceStatus(DeviceServiceImpl.java:318)
at com.extreme.common.poller.PollerUtils.updateDeviceStatus(PollerUtils.java:216)
at com.extreme.common.poller.SnmpStatusPollerTask.complete(SnmpStatusPollerTask.java:194)
at com.extreme.common.network.task.Task.complete(Task.java:553)
at com.extreme.common.network.request.RequestSnmp.processRequest(RequestSnmp.java:110)
at com.extreme.common.snmp.SnmpEngine$SnmpWrapper.doComplete(SnmpEngine.java:272)
at com.extreme.common.snmp.SnmpEngine$SnmpWrapper.access$300(SnmpEngine.java:156)
at com.extreme.common.snmp.SnmpEngine$SnmpWrapper$1.run(SnmpEngine.java:195)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

Thanks in advance!
Cheers,
Martin

[MartinS1]

The 1st backup takes about 3 minutes (~125MB), the 2nd takes about 51 minutes (~17GB)

Just before the finish of the second backup, there is a massive utilization of the disks.
This seems to be the problem which was not there before the update.

I'm going to open a GTAC-case 😕

regards,
Martin

[Frank_Thomas]

How big are your Database backups also how long do they take to complete? The completion time should be in console event logs.

[MartinS1]

I'm going to check tomorrow, as it's the system of a customer I don't have access to it every day 

[Jeremy_Gibbs]

I am no expert, but it sounds like your server might be running out of resources... Do you monitor the CPU / disk utilization at all?