Could this script interrupt SNMP process and cause high CPU utilization?

  • 0
  • 1
  • Problem
  • Updated 2 years ago
  • Solved
  • (Edited)
Hello, everybody!

Every morning I get the email from Netsight:

Netsight Critical Alarm: Device Down

Device: 192.168.251.161 (This is a stack of 5 switches)
Severity: Critical
Message: SNMP Contact Lost: No SNMP reply from device 192.168.251.161 caused by SNMP Error: Timeout [4098], last uptime was 10 Days

Could be it caused by these scripts? Especially the last one?

# Module upm configuration.#
create upm profile blackhole1
set var portb 0
set var portb $TCL(string range ${EVENT.LOG_PARAM_0} 36 39)
disable port $portb
create log entry "Port $portb blocked by UPM"
delete var portb
.
create upm profile mac_move
set var portb1 0
set var portb2 0
set var mac 0
set var portb1 $EVENT.LOG_PARAM_2
set var portb2 $EVENT.LOG_PARAM_3
set var mac $EVENT.LOG_PARAM_0
disable port $portb1
disable port $portb2
create log entry "Port $portb1 and $portb2 was disabled by UPM sript (mac_move), flapping MAC is $mac "
delete var portb1
delete var portb2
delete var mac
.
create upm profile tftp-save
enable cli scripting
set var CLI.OUT " "
show switch
set var base_list2 $TCL(split ${CLI.OUT} "\n")
set var baseSystemTime $TCL(clock seconds)
set var systemTime $TCL(clock format $baseSystemTime -format {%d.%m.%Y_%H.%M.%S})
set var systemname $TCL(lindex $base_list2 1)
set var systemname $TCL(string replace $systemname 0 17 {})
delete var "CLI.OUT"
#set var filename "$(systemname)_$(systemTime)"
set var filename "$(systemname)"
delete var systemname
#delete var systemtype
delete var systemTime
delete var baseSystemTime
delete var base_list2
upload config 192.168.251.1 cfg/$(filename).cfg VR-Default
delete var filename
.

#

Many thanks in advance,

Ilya
Photo of Ilya Semenov

Ilya Semenov

  • 4,610 Points 4k badge 2x thumb

Posted 2 years ago

  • 0
  • 1
Photo of Erik Auerswald

Erik Auerswald, Embassador

  • 12,960 Points 10k badge 2x thumb
Hi Ilya,

how long does the TFTP upload take? Does this run at the same time as the SNMP errors occur?

That script might cause the timeouts.

Br,
Erik
Photo of Ilya Semenov

Ilya Semenov

  • 4,610 Points 4k badge 2x thumb
I got the outputs:


Slot-7 sw-416-1.1 # sh upm timers
--------------------------------------------------------------------------------
UPM               Profile       Flags              Next Execution
Timer             Name                             time
--------------------------------------------------------------------------------
--------------------------------------------------------------------------------

Number of UPM Timers: 0
Flags: e - Profile is enabled, d: Profile is disabled
       o -Timer is non-periodic, p - Timer is periodic
Slot-7 sw-416-1.2 # sh upm history
--------------------------------------------------------------------------------
Exec  Event/              Profile          Port Status Time Launched
Id    Timer/ Log filter
--------------------------------------------------------------------------------
989   Log-Message(cfg_save tftp-save        --- Running 2016-10-21 14:04:16
988   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 14:04:05
987   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 14:03:52
986   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 14:03:41
985   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 14:03:29
984   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 14:03:17
983   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 14:03:06
982   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 14:02:55
981   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 14:02:43
980   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 14:02:30
979   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 14:02:15
978   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 14:01:59
977   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 14:01:47
976   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 14:01:35
975   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 14:01:23
974   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 14:01:12
973   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 14:01:00
972   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 14:00:49
971   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 14:00:38
970   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 14:00:26
969   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 14:00:14
968   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 14:00:02
967   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 13:59:51
966   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 13:59:39
965   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 13:59:27
964   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 13:59:16
963   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 13:59:04
962   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 13:58:53
961   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 13:58:42
960   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 13:58:30
959   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 13:58:18
958   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 13:58:07
957   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 13:57:55
956   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 13:57:44
955   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 13:57:32
954   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 13:57:20
953   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 13:57:08
952   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 13:56:57
951   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 13:56:45
950   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 13:56:33
949   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 13:56:21
948   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 13:56:06
947   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 13:55:53
946   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 13:55:41
945   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 13:55:30
944   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 13:55:18
943   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 13:55:07
942   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 13:54:55
941   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 13:54:43
940   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 13:54:32
939   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 13:54:20
938   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 13:54:08
937   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 13:53:56
936   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 13:53:44
935   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 13:53:32
934   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 13:53:20
933   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 13:53:09
932   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 13:52:57
931   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 13:52:45
930   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 13:52:34
929   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 13:52:23
928   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 13:52:11
927   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 13:51:59
926   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 13:51:47
925   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 13:51:36
924   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 13:51:24
923   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 13:51:13
922   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 13:51:01
921   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 13:50:50
920   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 13:50:38
919   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 13:50:26
918   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 13:50:15
917   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 13:50:03
916   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 13:49:52
915   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 13:49:40
914   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 13:49:28
913   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 13:49:16
912   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 13:49:05
911   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 13:48:53
910   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 13:48:42
909   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 13:48:30
908   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 13:48:18
907   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 13:48:07
906   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 13:47:54
905   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 13:47:43
904   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 13:47:31
903   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 13:47:19
902   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 13:47:08
901   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 13:46:56
900   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 13:46:44
899   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 13:46:33
898   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 13:46:21
897   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 13:46:10
896   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 13:45:58
895   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 13:45:47
894   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 13:45:35
893   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 13:45:23
892   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 13:45:11
891   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 13:45:00
890   Log-Message(cfg_save tftp-save        --- Fail    2016-10-21 13:44:48
--------------------------------------------------------------------------------

Number of UPM Events in Queue for execution: 1001
Slot-7 sw-416-1.3 #

Could you please tell me why has it gone to endless cycle?

Many thanks in advance,

Ilya
Photo of Grosjean, Stephane

Grosjean, Stephane, Employee

  • 12,850 Points 10k badge 2x thumb
Hi,


there're too many UPM events queued here, that can certainly cause problems. You should disable it first, to empty the queue and try to find what is not working in your case. Do a detailed view of one of the event to try to see the issue (why it fails).
Photo of Ilya Semenov

Ilya Semenov

  • 4,610 Points 4k badge 2x thumb
Hello, Erik!

Many thanks for your reply!

I am not sure whether it works at all... Now I check its start time. But I sure that its start time matches the fail time of SNMP. 

I have investigated this morning that if I try to archive configuration on this switch with Netsight inventory tool, it begin endless uploading the configuration file. It uploads the configuration again and again, the same file. I can delete it, and it copied again.

I've selected 3 switches, two copied its confs as it should be, and this one - went to endless cycle.

I wait for "show upm timer" and "show upm history" outputs from the customer at the moment. Hope, I will have additional information soon.

Thank you!
Photo of Erik Auerswald

Erik Auerswald, Embassador

  • 12,960 Points 10k badge 2x thumb
Hello Ilya,

I would suggest you open a GTAC case for this issue.

Br,
Erik