Switches lose connection to ExtremeControl 7.0 .4.29 (TAFKAN) while performing DB backup

MartinS1 — Wed, 24 Aug 2016 11:38:00 GMT

Hi there,

last week we decided to upgrade to NetSight Version 7.0.4.29. 😄

Since then, every time the NetSight DB is backuped, all switches appear as "Device Down" for a short period of time. But long enough for the system to send out the e-mail alarms.
As we've got 900+ Switches descovered, there are a lot of e-mails sent 😕

Does anyone have the same problem? Or should I open a GTAC case?!

This is the NetSight Error Log during the Upgrade:
2016-08-23 22:49:39,509 ERROR [com.extreme.common.poller.PollerUtils] processThrowable [3 updateDeviceStatus - Error sending device change messages] details:
com.enterasys.netsight.api.lockmgr.NSLockException: Could not acquire lock.
at com.enterasys.netsight.api.lockmgr.NSLock.acquire(NSLock.java:166)
at com.enterasys.netsight.server.ServerContext.acquireDeviceTreeWriteLock(ServerContext.java:750)
at com.enterasys.netsight.server.device.DeviceServiceImpl.updateObjectsInternal(DeviceServiceImpl.java:346)
at com.enterasys.netsight.server.device.DeviceServiceImpl.updateDeviceStatus(DeviceServiceImpl.java:318)
at com.extreme.common.poller.PollerUtils.updateDeviceStatus(PollerUtils.java:216)
at com.extreme.common.poller.SnmpStatusPollerTask.complete(SnmpStatusPollerTask.java:194)
at com.extreme.common.network.task.Task.complete(Task.java:553)
at com.extreme.common.network.request.RequestSnmp.processRequest(RequestSnmp.java:110)
at com.extreme.common.snmp.SnmpEngine$SnmpWrapper.doComplete(SnmpEngine.java:272)
at com.extreme.common.snmp.SnmpEngine$SnmpWrapper.access$300(SnmpEngine.java:156)
at com.extreme.common.snmp.SnmpEngine$SnmpWrapper$1.run(SnmpEngine.java:195)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2016-08-23 22:49:40,119 ERROR [com.extreme.common.poller.PollerUtils] processThrowable [2 updateDeviceStatus - Error sending device change messages] Message:Could not acquire lock. Cause:null

Called from:
Line:194 com.extreme.common.poller.SnmpStatusPollerTask method:complete
Line:553 com.extreme.common.network.task.Task method:complete
Line:110 com.extreme.common.network.request.RequestSnmp method:processRequest
Line:272 com.extreme.common.snmp.SnmpEngine$SnmpWrapper method:doComplete
Line:156 com.extreme.common.snmp.SnmpEngine$SnmpWrapper method:access$300
Line:195 com.extreme.common.snmp.SnmpEngine$SnmpWrapper$1 method:run
2016-08-23 22:49:40,119 ERROR [com.extreme.common.poller.PollerUtils] processThrowable [3 updateDeviceStatus - Error sending device change messages] Message:Could not acquire lock. Cause:null

Called from:
Line:194 com.extreme.common.poller.SnmpStatusPollerTask method:complete
Line:553 com.extreme.common.network.task.Task method:complete
Line:110 com.extreme.common.network.request.RequestSnmp method:processRequest
Line:272 com.extreme.common.snmp.SnmpEngine$SnmpWrapper method:doComplete
Line:156 com.extreme.common.snmp.SnmpEngine$SnmpWrapper method:access$300
Line:195 com.extreme.common.snmp.SnmpEngine$SnmpWrapper$1 method:run
2016-08-23 22:49:40,119 ERROR [com.extreme.common.poller.PollerUtils] processThrowable [2 updateDeviceStatus - Error sending device change messages] Message:Could not acquire lock. Cause:null

Called from:
Line:194 com.extreme.common.poller.SnmpStatusPollerTask method:complete
Line:553 com.extreme.common.network.task.Task method:complete
Line:110 com.extreme.common.network.request.RequestSnmp method:processRequest
Line:272 com.extreme.common.snmp.SnmpEngine$SnmpWrapper method:doComplete
Line:156 com.extreme.common.snmp.SnmpEngine$SnmpWrapper method:access$300
Line:195 com.extreme.common.snmp.SnmpEngine$SnmpWrapper$1 method:run
2016-08-23 22:49:40,119 ERROR [com.extreme.common.poller.PollerUtils] processThrowable [2 updateDeviceStatus - Error sending device change messages] Message:Could not acquire lock. Cause:null

Called from:
Line:194 com.extreme.common.poller.SnmpStatusPollerTask method:complete
Line:553 com.extreme.common.network.task.Task method:complete
Line:110 com.extreme.common.network.request.RequestSnmp method:processRequest
Line:272 com.extreme.common.snmp.SnmpEngine$SnmpWrapper method:doComplete
Line:156 com.extreme.common.snmp.SnmpEngine$SnmpWrapper method:access$300
Line:195 com.extreme.common.snmp.SnmpEngine$SnmpWrapper$1 method:run
2016-08-23 22:49:39,820 ERROR [org.jboss.mq.server.JMSDestinationManager] The connection to client ID:98 failed.
2016-08-23 22:49:39,509 ERROR [com.extreme.common.poller.PollerUtils] processThrowable [5 updateDeviceStatus - Error sending device change messages] details:
com.enterasys.netsight.api.lockmgr.NSLockException: Could not acquire lock.
at com.enterasys.netsight.api.lockmgr.NSLock.acquire(NSLock.java:166)
at com.enterasys.netsight.server.ServerContext.acquireDeviceTreeWriteLock(ServerContext.java:750)
at com.enterasys.netsight.server.device.DeviceServiceImpl.updateObjectsInternal(DeviceServiceImpl.java:346)
at com.enterasys.netsight.server.device.DeviceServiceImpl.updateDeviceStatus(DeviceServiceImpl.java:318)
at com.extreme.common.poller.PollerUtils.updateDeviceStatus(PollerUtils.java:216)
at com.extreme.common.poller.SnmpStatusPollerTask.complete(SnmpStatusPollerTask.java:194)
at com.extreme.common.network.task.Task.complete(Task.java:553)
at com.extreme.common.network.request.RequestSnmp.processRequest(RequestSnmp.java:110)
at com.extreme.common.snmp.SnmpEngine$SnmpWrapper.doComplete(SnmpEngine.java:272)
at com.extreme.common.snmp.SnmpEngine$SnmpWrapper.access$300(SnmpEngine.java:156)
at com.extreme.common.snmp.SnmpEngine$SnmpWrapper$1.run(SnmpEngine.java:195)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2016-08-23 22:49:39,509 ERROR [com.extreme.common.poller.PollerUtils] processThrowable [2 updateDeviceStatus - Error sending device change messages] details:
com.enterasys.netsight.api.lockmgr.NSLockException: Could not acquire lock.
at com.enterasys.netsight.api.lockmgr.NSLock.acquire(NSLock.java:166)
at com.enterasys.netsight.server.ServerContext.acquireDeviceTreeWriteLock(ServerContext.java:750)
at com.enterasys.netsight.server.device.DeviceServiceImpl.updateObjectsInternal(DeviceServiceImpl.java:346)
at com.enterasys.netsight.server.device.DeviceServiceImpl.updateDeviceStatus(DeviceServiceImpl.java:318)
at com.extreme.common.poller.PollerUtils.updateDeviceStatus(PollerUtils.java:216)
at com.extreme.common.poller.SnmpStatusPollerTask.complete(SnmpStatusPollerTask.java:194)
at com.extreme.common.network.task.Task.complete(Task.java:553)
at com.extreme.common.network.request.RequestSnmp.processRequest(RequestSnmp.java:110)
at com.extreme.common.snmp.SnmpEngine$SnmpWrapper.doComplete(SnmpEngine.java:272)
at com.extreme.common.snmp.SnmpEngine$SnmpWrapper.access$300(SnmpEngine.java:156)
at com.extreme.common.snmp.SnmpEngine$SnmpWrapper$1.run(SnmpEngine.java:195)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2016-08-23 22:49:39,509 ERROR [com.extreme.common.poller.PollerUtils] processThrowable [2 updateDeviceStatus - Error sending device change messages] details:
com.enterasys.netsight.api.lockmgr.NSLockException: Could not acquire lock.
at com.enterasys.netsight.api.lockmgr.NSLock.acquire(NSLock.java:166)
at com.enterasys.netsight.server.ServerContext.acquireDeviceTreeWriteLock(ServerContext.java:750)
at com.enterasys.netsight.server.device.DeviceServiceImpl.updateObjectsInternal(DeviceServiceImpl.java:346)
at com.enterasys.netsight.server.device.DeviceServiceImpl.updateDeviceStatus(DeviceServiceImpl.java:318)
at com.extreme.common.poller.PollerUtils.updateDeviceStatus(PollerUtils.java:216)
at com.extreme.common.poller.SnmpStatusPollerTask.complete(SnmpStatusPollerTask.java:194)
at com.extreme.common.network.task.Task.complete(Task.java:553)
at com.extreme.common.network.request.RequestSnmp.processRequest(RequestSnmp.java:110)
at com.extreme.common.snmp.SnmpEngine$SnmpWrapper.doComplete(SnmpEngine.java:272)
at com.extreme.common.snmp.SnmpEngine$SnmpWrapper.access$300(SnmpEngine.java:156)
at com.extreme.common.snmp.SnmpEngine$SnmpWrapper$1.run(SnmpEngine.java:195)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2016-08-23 22:49:39,509 ERROR [com.extreme.common.poller.PollerUtils] processThrowable [4 updateDeviceStatus - Error sending device change messages] details:
com.enterasys.netsight.api.lockmgr.NSLockException: Could not acquire lock.
at com.enterasys.netsight.api.lockmgr.NSLock.acquire(NSLock.java:166)
at com.enterasys.netsight.server.ServerContext.acquireDeviceTreeWriteLock(ServerContext.java:750)
at com.enterasys.netsight.server.device.DeviceServiceImpl.updateObjectsInternal(DeviceServiceImpl.java:346)
at com.enterasys.netsight.server.device.DeviceServiceImpl.updateDeviceStatus(DeviceServiceImpl.java:318)
at com.extreme.common.poller.PollerUtils.updateDeviceStatus(PollerUtils.java:216)
at com.extreme.common.poller.SnmpStatusPollerTask.complete(SnmpStatusPollerTask.java:194)
at com.extreme.common.network.task.Task.complete(Task.java:553)
at com.extreme.common.network.request.RequestSnmp.processRequest(RequestSnmp.java:110)
at com.extreme.common.snmp.SnmpEngine$SnmpWrapper.doComplete(SnmpEngine.java:272)
at com.extreme.common.snmp.SnmpEngine$SnmpWrapper.access$300(SnmpEngine.java:156)
at com.extreme.common.snmp.SnmpEngine$SnmpWrapper$1.run(SnmpEngine.java:195)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

Thanks in advance!
Cheers,
Martin

RE: Switches lose connection to ExtremeControl 7.0 .4.29 (TAFKAN) while performing DB backup

Jeremy_Gibbs — Wed, 24 Aug 2016 18:17:00 GMT

I am no expert, but it sounds like your server might be running out of resources... Do you monitor the CPU / disk utilization at all?

RE: Switches lose connection to ExtremeControl 7.0 .4.29 (TAFKAN) while performing DB backup

Frank_Thomas — Wed, 24 Aug 2016 23:16:00 GMT

How big are your Database backups also how long do they take to complete? The completion time should be in console event logs.

RE: Switches lose connection to ExtremeControl 7.0 .4.29 (TAFKAN) while performing DB backup

MartinS1 — Wed, 24 Aug 2016 23:16:00 GMT

I'm going to check tomorrow, as it's the system of a customer I don't have access to it every day 

RE: Switches lose connection to ExtremeControl 7.0 .4.29 (TAFKAN) while performing DB backup

MartinS1 — Tue, 30 Aug 2016 11:49:00 GMT

The 1st backup takes about 3 minutes (~125MB), the 2nd takes about 51 minutes (~17GB)

Just before the finish of the second backup, there is a massive utilization of the disks.
This seems to be the problem which was not there before the update.

I'm going to open a GTAC-case 😕

regards,
Martin

RE: Switches lose connection to ExtremeControl 7.0 .4.29 (TAFKAN) while performing DB backup

Tom_Currier — Thu, 01 Sep 2016 19:39:00 GMT

Martin;

I'm taking this case and will respond directly from there. As mentioned in prior threads, this appears to be a resource issue where we might be able to make some adjustments to memory allocation to solve this issue.

Tom

RE: Switches lose connection to ExtremeControl 7.0 .4.29 (TAFKAN) while performing DB backup

MartinS1 — Wed, 14 Sep 2016 16:33:00 GMT

Update:
Netsight is still loosing connections during DB Updates.
The Memory Usages is currently set to ~12GB, and the mySQL is using "my-huge.ini".

RE: Switches lose connection to ExtremeControl 7.0 .4.29 (TAFKAN) while performing DB backup

Tom_Currier — Wed, 21 Sep 2016 19:43:00 GMT

The GTAC provided an Enterprise Backup solution for version 7.0.5.12 which should alleviate this problem.

RE: Switches lose connection to ExtremeControl 7.0 .4.29 (TAFKAN) while performing DB backup

MartinS1 — Fri, 30 Sep 2016 18:27:00 GMT

Hi Guys,

GTAC provided us the guide for "Enterprise DB Update".

It works pretty well 🙂

Thanks!

topic RE: Switches lose connection to ExtremeControl 7.0 .4.29 (TAFKAN) while performing DB backup in ExtremeCloud IQ- Site Engine Management Center

Switches lose connection to ExtremeControl 7.0 .4.29 (TAFKAN) while performing DB backup

RE: Switches lose connection to ExtremeControl 7.0 .4.29 (TAFKAN) while performing DB backup

RE: Switches lose connection to ExtremeControl 7.0 .4.29 (TAFKAN) while performing DB backup

RE: Switches lose connection to ExtremeControl 7.0 .4.29 (TAFKAN) while performing DB backup

RE: Switches lose connection to ExtremeControl 7.0 .4.29 (TAFKAN) while performing DB backup

RE: Switches lose connection to ExtremeControl 7.0 .4.29 (TAFKAN) while performing DB backup

RE: Switches lose connection to ExtremeControl 7.0 .4.29 (TAFKAN) while performing DB backup

RE: Switches lose connection to ExtremeControl 7.0 .4.29 (TAFKAN) while performing DB backup

RE: Switches lose connection to ExtremeControl 7.0 .4.29 (TAFKAN) while performing DB backup