Could this script interrupt SNMP process and cause high CPU utilization?

Ilya_Semenov — Thu, 20 Oct 2016 13:46:00 GMT

Hello, everybody!

Every morning I get the email from Netsight:

Netsight Critical Alarm: Device Down

Device: 192.168.251.161 (This is a stack of 5 switches)
Severity: Critical
Message: SNMP Contact Lost: No SNMP reply from device 192.168.251.161 caused by SNMP Error: Timeout [4098], last uptime was 10 Days

Could be it caused by these scripts? Especially the last one?

# Module upm configuration.#
create upm profile blackhole1
set var portb 0
set var portb $TCL(string range ${EVENT.LOG_PARAM_0} 36 39)
disable port $portb
create log entry "Port $portb blocked by UPM"
delete var portb
.
create upm profile mac_move
set var portb1 0
set var portb2 0
set var mac 0
set var portb1 $EVENT.LOG_PARAM_2
set var portb2 $EVENT.LOG_PARAM_3
set var mac $EVENT.LOG_PARAM_0
disable port $portb1
disable port $portb2
create log entry "Port $portb1 and $portb2 was disabled by UPM sript (mac_move), flapping MAC is $mac "
delete var portb1
delete var portb2
delete var mac
.
create upm profile tftp-save
enable cli scripting
set var CLI.OUT " "
show switch
set var base_list2 $TCL(split ${CLI.OUT} "\n")
set var baseSystemTime $TCL(clock seconds)
set var systemTime $TCL(clock format $baseSystemTime -format {%d.%m.%Y_%H.%M.%S})
set var systemname $TCL(lindex $base_list2 1)
set var systemname $TCL(string replace $systemname 0 17 {})
delete var "CLI.OUT"
#set var filename "$(systemname)_$(systemTime)"
set var filename "$(systemname)"
delete var systemname
#delete var systemtype
delete var systemTime
delete var baseSystemTime
delete var base_list2
upload config 192.168.251.1 cfg/$(filename).cfg VR-Default
delete var filename
.

#

Many thanks in advance,

Ilya

RE: Could this script interrupt SNMP process and cause high CPU utilization?

Erik_Auerswald — Fri, 21 Oct 2016 15:06:00 GMT

Hi Ilya,

how long does the TFTP upload take? Does this run at the same time as the SNMP errors occur?

That script might cause the timeouts.

Br,
Erik

RE: Could this script interrupt SNMP process and cause high CPU utilization?

Ilya_Semenov — Fri, 21 Oct 2016 15:06:00 GMT

I got the outputs:

Slot-7 sw-416-1.1 # sh upm timers
--------------------------------------------------------------------------------
UPM Profile Flags Next Execution
Timer Name time
--------------------------------------------------------------------------------
--------------------------------------------------------------------------------

Number of UPM Timers: 0
Flags: e - Profile is enabled,  Profile is disabled
o -Timer is non-periodic, p - Timer is periodic
Slot-7 sw-416-1.2 # sh upm history
--------------------------------------------------------------------------------
Exec Event/ Profile Port Status Time Launched
Id Timer/ Log filter
--------------------------------------------------------------------------------
989 Log-Message(cfg_save tftp-save --- Running 2016-10-21 14:04:16
988 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 14:04:05
987 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 14:03:52
986 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 14:03:41
985 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 14:03:29
984 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 14:03:17
983 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 14:03:06
982 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 14:02:55
981 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 14:02:43
980 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 14:02:30
979 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 14:02:15
978 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 14:01:59
977 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 14:01:47
976 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 14:01:35
975 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 14:01:23
974 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 14:01:12
973 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 14:01:00
972 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 14:00:49
971 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 14:00:38
970 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 14:00:26
969 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 14:00:14
968 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 14:00:02
967 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 13:59:51
966 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 13:59:39
965 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 13:59:27
964 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 13:59:16
963 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 13:59:04
962 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 13:58:53
961 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 13:58:42
960 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 13:58:30
959 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 13:58:18
958 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 13:58:07
957 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 13:57:55
956 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 13:57:44
955 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 13:57:32
954 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 13:57:20
953 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 13:57:08
952 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 13:56:57
951 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 13:56:45
950 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 13:56:33
949 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 13:56:21
948 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 13:56:06
947 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 13:55:53
946 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 13:55:41
945 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 13:55:30
944 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 13:55:18
943 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 13:55:07
942 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 13:54:55
941 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 13:54:43
940 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 13:54:32
939 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 13:54:20
938 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 13:54:08
937 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 13:53:56
936 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 13:53:44
935 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 13:53:32
934 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 13:53:20
933 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 13:53:09
932 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 13:52:57
931 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 13:52:45
930 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 13:52:34
929 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 13:52:23
928 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 13:52:11
927 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 13:51:59
926 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 13:51:47
925 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 13:51:36
924 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 13:51:24
923 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 13:51:13
922 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 13:51:01
921 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 13:50:50
920 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 13:50:38
919 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 13:50:26
918 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 13:50:15
917 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 13:50:03
916 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 13:49:52
915 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 13:49:40
914 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 13:49:28
913 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 13:49:16
912 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 13:49:05
911 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 13:48:53
910 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 13:48:42
909 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 13:48:30
908 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 13:48:18
907 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 13:48:07
906 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 13:47:54
905 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 13:47:43
904 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 13:47:31
903 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 13:47:19
902 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 13:47:08
901 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 13:46:56
900 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 13:46:44
899 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 13:46:33
898 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 13:46:21
897 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 13:46:10
896 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 13:45:58
895 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 13:45:47
894 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 13:45:35
893 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 13:45:23
892 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 13:45:11
891 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 13:45:00
890 Log-Message(cfg_save tftp-save --- Fail 2016-10-21 13:44:48
--------------------------------------------------------------------------------

Number of UPM Events in Queue for execution: 1001
Slot-7 sw-416-1.3 #

Could you please tell me why has it gone to endless cycle?

Many thanks in advance,

Ilya

RE: Could this script interrupt SNMP process and cause high CPU utilization?

Stephane_Grosj1 — Fri, 21 Oct 2016 15:06:00 GMT

Hi,

there're too many UPM events queued here, that can certainly cause problems. You should disable it first, to empty the queue and try to find what is not working in your case. Do a detailed view of one of the event to try to see the issue (why it fails).

RE: Could this script interrupt SNMP process and cause high CPU utilization?

Ilya_Semenov — Fri, 21 Oct 2016 16:13:00 GMT

Hello, Erik!

Many thanks for your reply!

I am not sure whether it works at all... Now I check its start time. But I sure that its start time matches the fail time of SNMP.

I have investigated this morning that if I try to archive configuration on this switch with Netsight inventory tool, it begin endless uploading the configuration file. It uploads the configuration again and again, the same file. I can delete it, and it copied again.

I've selected 3 switches, two copied its confs as it should be, and this one - went to endless cycle.

I wait for "show upm timer" and "show upm history" outputs from the customer at the moment. Hope, I will have additional information soon.

Thank you!

RE: Could this script interrupt SNMP process and cause high CPU utilization?

Erik_Auerswald — Mon, 24 Oct 2016 12:11:00 GMT

Hello Ilya,

I would suggest you open a GTAC case for this issue.

Br,
Erik

topic RE: Could this script interrupt SNMP process and cause high CPU utilization? in ExtremeCloud IQ- Site Engine Management Center

Could this script interrupt SNMP process and cause high CPU utilization?

RE: Could this script interrupt SNMP process and cause high CPU utilization?

RE: Could this script interrupt SNMP process and cause high CPU utilization?

RE: Could this script interrupt SNMP process and cause high CPU utilization?

RE: Could this script interrupt SNMP process and cause high CPU utilization?

RE: Could this script interrupt SNMP process and cause high CPU utilization?