high CPU in X450 switch

[hasan_issa]

Hi everybody



we have X450 switch in our network is facing high CPU in the following process:



1- ./hal

2- ./FDB

3- bcmRX



also when I ping to the IP of the switch I noted the bcmRX process increase

I need to Know if this normal?



Mem: 246820K used, 7816K free, 0K shrd, 32580K buff, 72532K cached

CPU: 0.0% usr 100% sys 0.0% nic 0.0% idle 0.0% io 0.0% irq 0.0% sirq

Load average: 4.11 4.11 4.08 3/182 2021

PID PPID USER STAT RSS %MEM CPU %CPU COMMAND

1433 1 root S 3316 1.3 0 26.3 ./fdb

1405 1 root S < 11140 4.3 0 21.0 ./hal

1261 2 root SW 0 0.0 0 21.0 [bcmRX]

2021 2020 root R 780 0.3 0 15.7 top -d 3

1474 1 root S 1920 0.7 0 5.2 ./bfd

1407 1 root S 1596 0.6 0 5.2 ./nodemgr

1506 1505 root S 764 0.3 0 5.2 ./telnetd -e

1409 1 root S 31584 12.3 0 0.0 ./cliMaster

1415 1 root S 5456 2.1 0 0.0 ./snmpSubagent

1536 1 root S 4688 1.8 0 0.0 ./xmld

1561 1 root S 3964 1.5 0 0.0 ./idMgr

1514 1 root S 3920 1.5 0 0.0 ./etmon

1401 1 root S 3732 1.4 0 0.0 ./emsServer

1472 1 root S 3464 1.3 0 0.0 ./mcmgr

1413 1 root S 3292 1.2 0 0.0 ./snmpMaster

1423 1 root S 3288 1.2 0 0.0 ./vlan

1492 1 root S 3076 1.2 0 0.0 ./pim

1466 1 root S 2916 1.1 0 0.0 ./rtmgr update

1411 1 root S 2848 1.1 0 0.0 ./cfgmgr

1501 1 root S 2688 1.0 0 0.0 ./netTools

1499 1 root S 2640 1.0 0 0.0 ./acl



regards,

Hasan

[Prashanth_KG]

Hi Hasan,

Thanks a lot for your effort in collecting the requested outputs.
I see that you have added the log counters. So, if you issue the command show log, you will be able to know what mac addresses are added and deleted along with the port numbers and VLAN information.
See if you can find any pattern of a specific port or any specific mac-addresses. that should help a bit.

If you are using STP or any L2 loop prevention protocol, please check if there are any frequent topology changes. This could result in the FDB flush and forced re-learning.

Regarding the impact, it is hard to say unless we know the network completely. However it is not recommended to have high CPU in a switch. So, it would be good to sort this out.

I think this would be the right time to open a GTAC case with all this information as Brandon and Patrick suggested!

Thanks!

[hasan_issa]

Hi Prashanth,


thanks for your support

I have configured the log for MAC-add , MAC-del and MAC-mov in the switch and I found a lot of packet related for add and delete MACs



below the log of FDB counters:





JUB_020.22 # show log counters FDB occurred

Component SubComponent Condition Severity Occurred I Ntfd

----------- ------------ ----------------------- ------------- -------- - ----

FDB ArpDebugSummary Debug-Summary 14638 N 0

FDB FdbDebugSummary Debug-Summary 2067684 N 0

FDB FilterDebugSummary Debug-Summary 9 N 0

FDB MACTracking MACAdd Notice 51 Y 47

FDB MACTracking MACDel Notice 49 Y 46





I need to know we can prevent add and delete MACs in Vlans also I need to know a huge of adding and deleting MACs from Vlans will affect the switch





As your request in the Below the log of top command before press number 1 and during press number 1



Before press number 1



Mem: 247840K used, 6796K free, 0K shrd, 32804K buff, 73132K cached

CPU: 2.6% usr 1.9% sys 0.0% nic 94.3% idle 0.0% io 0.3% irq 0.6% sirq

Load average: 4.04 4.09 4.08 3/182 2301

PID PPID USER STAT RSS %MEM CPU %CPU COMMAND

1433 1 root S 3324 1.3 0 1.9 ./fdb

1405 1 root S < 11432 4.4 0 0.9 ./hal

1490 1 root S 2096 0.8 0 0.6 ./r.png

2301 2300 root R 780 0.3 0 0.6 top -d 3

1413 1 root S 3296 1.2 0 0.3 ./snmpMaster

1407 1 root S 1660 0.6 0 0.3 ./nodemgr

1261 2 root SW 0 0.0 0 0.3 [bcmRX]

1409 1 root S 31460 12.2 0 0.0 ./cliMaster

1415 1 root S 5460 2.1 0 0.0 ./snmpSubagent

1536 1 root S 4688 1.8 0 0.0 ./xmld

1561 1 root S 3964 1.5 0 0.0 ./idMgr

1514 1 root S 3920 1.5 0 0.0 ./etmon

1401 1 root S 3732 1.4 0 0.0 ./emsServer

1472 1 root S 3484 1.3 0 0.0 ./mcmgr

1423 1 root S 3336 1.3 0 0.0 ./vlan

1492 1 root S 3076 1.2 0 0.0 ./pim

1466 1 root S 2936 1.1 0 0.0 ./rtmgr update

1411 1 root S 2852 1.1 0 0.0 ./cfgmgr

1501 1 root S 2688 1.0 0 0.0 ./netTools

1499 1 root S 2640 1.0 0 0.0 ./acl

1486 1 root S 2640 1.0 0 0.0 ./ospfv3





during press number 1





Mem: 247840K used, 6796K free, 0K shrd, 32804K buff, 73132K cached

CPU: 0.0% usr 100% sys 0.0% nic 0.0% idle 0.0% io 0.0% irq 0.0% sirq

Load average: 4.18 4.14 4.10 3/182 2301

PID PPID USER STAT RSS %MEM CPU %CPU COMMAND

1433 1 root S 3324 1.3 0 30.4 ./fdb

2301 2300 root R 780 0.3 0 26.0 top -d 3

1405 1 root S < 11432 4.4 0 21.7 ./hal

1492 1 root S 3076 1.2 0 8.6 ./pim

1261 2 root RW 0 0.0 0 8.6 [bcmRX]

1472 1 root S 3484 1.3 0 4.3 ./mcmgr

1409 1 root S 31460 12.2 0 0.0 ./cliMaster

1415 1 root S 5460 2.1 0 0.0 ./snmpSubagent

1536 1 root S 4688 1.8 0 0.0 ./xmld

1561 1 root S 3964 1.5 0 0.0 ./idMgr

1514 1 root S 3920 1.5 0 0.0 ./etmon

1401 1 root S 3732 1.4 0 0.0 ./emsServer

1423 1 root S 3336 1.3 0 0.0 ./vlan

1413 1 root S 3296 1.2 0 0.0 ./snmpMaster

1466 1 root S 2936 1.1 0 0.0 ./rtmgr update

1411 1 root S 2852 1.1 0 0.0 ./cfgmgr

1501 1 root S 2688 1.0 0 0.0 ./netTools

1499 1 root S 2640 1.0 0 0.0 ./acl

1486 1 root S 2640 1.0 0 0.0 ./ospfv3

1056 1 root S 2624 1.0 0 0.0 /exos/bin/epm -t 40 -f /exos/confi

1484 1 root S 2576 1.0 0 0.0 ./ospf



Regards,

Hasan

[Prashanth_KG]

Hi Hasan,

Looking at the l2stats output, the count of packets learned has increased which is an indication of the too much learning in the switch. I still bet on the mac-movement happening in the switch or too much addition and deletion of the entries happening in the switch.

example: Total number of packets learned = 391638.

As requested before, collect show log counters fdb occurred. Check for the counters of FDB.macadd, FDB.macdel and FDB.macmove.

Utilise the link below to configure mac-tracking and to identify the mac learning in the switch.

https://gtacknowledge.extremenetworks.com/articles/How_To/How-to-configure-MAC-tracking-in-EXOS/

Regarding your question regarding the top, can you please share the output while pressing 1 and without pressing 1 for our understanding.

Hope this helps!!
Keep us updated.

[Patrick_Voss]

l2stats shows a count since the uptime on the switch or the last time someone cleared the counters (This information can't be found). Packets going to the CPU is normal. The problem is when you have a bunch of packets going to the CPU constantly.

I would recommend contacting GTAC moving forward considering there might need to be some diagnostic steps they can take that will require debug mode.