x450 xOS16.1.3.6 CLI CPU Load

  • 0
  • 1
  • Question
  • Updated 1 year ago
  • Answered
  • (Edited)
Hello everyone,

First questing on this community as I've searched for a while and couldn't really find an answer.

I manage a set of x450 switches which are all running xOS16.1.3.6.

Sadly we don't have TFTP or ssh/sftp insfrastruture to be able to copy the primary.cf config file so the current backup solution is made with bash+expect sending sh conf detail.

Recently I've found out that from time to time a traps is sent refering to High CPU usage from the CLI process which is always right after the command is sent.

A few prints:
06/08/2017 13:17:29.14 <Warn:EPM.cpu> CPU utilization monitor: process cli consumes 92 % CPU

#######
# show log counters cli
Component   SubComponent Condition               Severity      Occurred I Ntfd
----------- ------------ ----------------------- ------------- -------- - ----
cli                      clearflowExecError      Error                0 Y    0
cli                      cliOutError             Error                0 Y    0
cli                      logLocalCmd             Info                 0 Y    0
cli                      logRemoteCmd            Info                81 Y  162
cli                      TermScrptTmout          Warning              0 Y    0
cli         Legacy       InstlModAltPartOK       Debug-Summary        0 N    0
cli         Legacy       InstlModCurPartOK       Debug-Summary        0 N    0
cli         Legacy       LcnsNotEnbl             Debug-Summary        0 N    0
cli         Legacy       ModInstl                Info                 0 Y    0
cli         Legacy       ModNotInstl             Info                 0 Y    0
cli         Legacy       UninstlModAltPartOK     Debug-Summary        0 N    0
cli         Legacy       UninstlModCurPartOK     Debug-Summary        0 N    0
cli         shell        completionBegin         Debug-Verbose       61 N    0
cli         shell        completionEnd           Debug-Verbose       61 N    0
cli         shell        completionRecv          Debug-Verbose       61 N    0
cli         shell        connect                 Debug-Summary        0 N    0
cli         shell        disconnect              Notice               0 N    0
cli         shell        executionBegin          Debug-Summary      660 N    0
cli         shell        executionEnd            Debug-Summary      659 N    0
cli         shell        executionRecv           Debug-Summary      660 N    0
cli         shell        idleTimeout             Notice               0 N    0
cli         subagent     connect                 Debug-Summary        0 N    0
cli         subagent     disconnect              Debug-Summary        0 N    0

#########
# sh log events CLI
Component   SubComponent Condition               Severity      Parameters
----------- ------------ ----------------------- ------------- ----------
cli                      clearflowExecError      Error          2 Total
cli                      cliOutError             Error          1 Total
cli                      logLocalCmd             Info           3 Total
cli                      logRemoteCmd            Info           4 Total
cli                      TermScrptTmout          Warning        2 Total
cli         Legacy       InstlModAltPartOK       Debug-Summary  1 Total
cli         Legacy       InstlModCurPartOK       Debug-Summary  1 Total
cli         Legacy       LcnsNotEnbl             Debug-Summary  0 Total
cli         Legacy       ModInstl                Info           0 Total
cli         Legacy       ModNotInstl             Info           0 Total
cli         Legacy       UninstlModAltPartOK     Debug-Summary  1 Total
cli         Legacy       UninstlModCurPartOK     Debug-Summary  1 Total
cli         shell        completionBegin         Debug-Verbose  4 Total
cli         shell        completionEnd           Debug-Verbose  4 Total
cli         shell        completionRecv          Debug-Verbose  2 Total
cli         shell        connect                 Debug-Summary  1 Total
cli         shell        disconnect              Notice         1 Total
cli         shell        executionBegin          Debug-Summary  4 Total
cli         shell        executionEnd            Debug-Summary  4 Total
cli         shell        executionRecv           Debug-Summary  2 Total
cli         shell        idleTimeout             Notice         3 Total
cli         subagent     connect                 Debug-Summary  2 Total
cli         subagent     disconnect              Debug-Summary  2 Total

######
# sh cpu proc cli

      CPU Utilization Statistics - Monitored every 5 seconds
-----------------------------------------------------------------------

Process      5   10   30   1    5    30   1    Max           Total
            secs secs secs min  mins mins hour            User/System
            util util util util util util util util       CPU Usage
            (%)  (%)  (%)  (%)   (%)  (%)  (%)  (%)         (secs)
-----------------------------------------------------------------------

System       10.9  9.5  9.3 10.3 12.3 11.3 14.4 99.9 41215.86   2547170.27
cli           0.0 15.8 14.5  1.2  6.8  5.9  3.0 92.6   636.23     235.08
######

Is this behavior expected & normal?

Thank you
Hugo
Photo of Hugo Gonçalves

Hugo Gonçalves

  • 82 Points 75 badge 2x thumb
  • confused

Posted 1 year ago

  • 0
  • 1
Photo of EtherMAN

EtherMAN, Embassador

  • 7,200 Points 5k badge 2x thumb
Yes and Yes.  Remember most of what your switches are doing is in hardware so CPU spikes only affect processes that may be dependent on the cpu.  We have a partner that uses Rancid and captures configs by doing not only a show config but a show config detail.  Doing that double command to display configuration and detail configuration always sets off CPU traps.  With that being said we have at least got them to run those backups during off hours when traffic patterns and usage is at a minimal.  Only you will know whither having high cpu usage is affecting your service delivery through the switches.  It all will depend on your specific usage and the timing of the spikes. 
Photo of Hugo Gonçalves

Hugo Gonçalves

  • 82 Points 75 badge 2x thumb
Hi EtherMAN,

Thank you for your reply.

In the above this has happened during day as I was trying replicate the 'issue'.

I have weekly backups running each Monday at 6am whereas the traffic is quite low at that time, anyway what sort of prints/commands would you advise to send to the switch to confirm no service outage/impact is present at the time of the 'backup'?

Worth mention that the switch is purely doing L2/L3 switching, no mpls is enabled and only one VR with 8 vlans with an 'external' port connected to a Cisco ASR9k which does all the mpls stuff across the network.

Regards,
Hugo