x450 xOS16.1.3.6 CLI CPU Load

[Hugo_Gonçalves]

Hello everyone,

First questing on this community as I've searched for a while and couldn't really find an answer.

I manage a set of x450 switches which are all running xOS16.1.3.6.

Sadly we don't have TFTP or ssh/sftp insfrastruture to be able to copy the primary.cf config file so the current backup solution is made with bash+expect sending sh conf detail.

Recently I've found out that from time to time a traps is sent refering to High CPU usage from the CLI process which is always right after the command is sent.

A few prints:
06/08/2017 13:17:29.14 CPU utilization monitor: process cli consumes 92 % CPU

#######
# show log counters cli
Component SubComponent Condition Severity Occurred I Ntfd
----------- ------------ ----------------------- ------------- -------- - ----
cli clearflowExecError Error 0 Y 0
cli cliOutError Error 0 Y 0
cli logLocalCmd Info 0 Y 0
cli logRemoteCmd Info 81 Y 162
cli TermScrptTmout Warning 0 Y 0
cli Legacy InstlModAltPartOK Debug-Summary 0 N 0
cli Legacy InstlModCurPartOK Debug-Summary 0 N 0
cli Legacy LcnsNotEnbl Debug-Summary 0 N 0
cli Legacy ModInstl Info 0 Y 0
cli Legacy ModNotInstl Info 0 Y 0
cli Legacy UninstlModAltPartOK Debug-Summary 0 N 0
cli Legacy UninstlModCurPartOK Debug-Summary 0 N 0
cli shell completionBegin Debug-Verbose 61 N 0
cli shell completionEnd Debug-Verbose 61 N 0
cli shell completionRecv Debug-Verbose 61 N 0
cli shell connect Debug-Summary 0 N 0
cli shell disconnect Notice 0 N 0
cli shell executionBegin Debug-Summary 660 N 0
cli shell executionEnd Debug-Summary 659 N 0
cli shell executionRecv Debug-Summary 660 N 0
cli shell idleTimeout Notice 0 N 0
cli subagent connect Debug-Summary 0 N 0
cli subagent disconnect Debug-Summary 0 N 0

#########
# sh log events CLI
Component SubComponent Condition Severity Parameters
----------- ------------ ----------------------- ------------- ----------
cli clearflowExecError Error 2 Total
cli cliOutError Error 1 Total
cli logLocalCmd Info 3 Total
cli logRemoteCmd Info 4 Total
cli TermScrptTmout Warning 2 Total
cli Legacy InstlModAltPartOK Debug-Summary 1 Total
cli Legacy InstlModCurPartOK Debug-Summary 1 Total
cli Legacy LcnsNotEnbl Debug-Summary 0 Total
cli Legacy ModInstl Info 0 Total
cli Legacy ModNotInstl Info 0 Total
cli Legacy UninstlModAltPartOK Debug-Summary 1 Total
cli Legacy UninstlModCurPartOK Debug-Summary 1 Total
cli shell completionBegin Debug-Verbose 4 Total
cli shell completionEnd Debug-Verbose 4 Total
cli shell completionRecv Debug-Verbose 2 Total
cli shell connect Debug-Summary 1 Total
cli shell disconnect Notice 1 Total
cli shell executionBegin Debug-Summary 4 Total
cli shell executionEnd Debug-Summary 4 Total
cli shell executionRecv Debug-Summary 2 Total
cli shell idleTimeout Notice 3 Total
cli subagent connect Debug-Summary 2 Total
cli subagent disconnect Debug-Summary 2 Total

######
# sh cpu proc cli

CPU Utilization Statistics - Monitored every 5 seconds
-----------------------------------------------------------------------

Process 5 10 30 1 5 30 1 Max Total
secs secs secs min mins mins hour User/System
util util util util util util util util CPU Usage
(%) (%) (%) (%) (%) (%) (%) (%) (secs)
-----------------------------------------------------------------------

System 10.9 9.5 9.3 10.3 12.3 11.3 14.4 99.9 41215.86 2547170.27
cli 0.0 15.8 14.5 1.2 6.8 5.9 3.0 92.6 636.23 235.08
######

Is this behavior expected & normal?

Thank you
Hugo

[Hugo_Gonçalves]

Hi EtherMAN, Thank you for your reply. In the above this has happened during day as I was trying replicate the 'issue'. I have weekly backups running each Monday at 6am whereas the traffic is quite low at that time, anyway what sort of prints/commands would you advise to send to the switch to confirm no service outage/impact is present at the time of the 'backup'? Worth mention that the switch is purely doing L2/L3 switching, no mpls is enabled and only one VR with 8 vlans with an 'external' port connected to a Cisco ASR9k which does all the mpls stuff across the network. Regards, Hugo

[EtherMAN]

Yes and Yes. Remember most of what your switches are doing is in hardware so CPU spikes only affect processes that may be dependent on the cpu. We have a partner that uses Rancid and captures configs by doing not only a show config but a show config detail. Doing that double command to display configuration and detail configuration always sets off CPU traps. With that being said we have at least got them to run those backups during off hours when traffic patterns and usage is at a minimal. Only you will know whither having high cpu usage is affecting your service delivery through the switches. It all will depend on your specific usage and the timing of the spikes.