High bcmRX process in Black Diamonds

  • 0
  • 1
  • Problem
  • Updated 4 years ago
  • Solved
We've been experiencing high bcmRX on our Black Diamonds and it seems to be impacting overall network performance. The bcmRX process starts running at about 15-20% (top command) on peak traffic hours. After taking a TCP-Dump on the Black Diamonds during low & high traffic times we saw that the main difference between both times is that during high load is that we saw a lot of TCP Re-transmissions and Dup ACK packets in Wireshark. During normal & low load times, our TCP Dumps are clean (no TCP Dups or re-transmissions).

Does anybody have any idea as to why we might be seeing this behavior? We're running ver 15.3.4.6 v1534b6-patch1-5 but we've also seen this with other versions.

Thank you.
Photo of Alex

Alex

  • 282 Points 250 badge 2x thumb

Posted 4 years ago

  • 0
  • 1
Photo of Alex

Alex

  • 282 Points 250 badge 2x thumb
Official Response
Ok, I can say that the issue is now solved (the bcmXR process is now hovering at ~2-5 % at peak hours). I'll try to describe it concisely below.

Problem: our network design was not very scalable and while it was sound a few years ago, it didn't take into account network growth and rapid wireless explosion. Thus, we had our 2 BDs connected to dozens of 650s (one 650 per campus building) which in turn where connected to 450s. In our design, our BDs where acting as the DG for all campus VLANs/subnets (around 250 VLANs) and thus, they were seeing all IPs on campus and adding them to the IP table. This caused many of these to be processed at the CPU when at peak traffic times.

Solution: we offloaded the routing as much as possible to the 650s on all campus buildings and thus, these 650s became the DGs for each building. After moving routing further away from the BDs, we saw an immediate impact in performance as these boxes didn't have to learn as many L3 addresses and consequently these packets could be sent the 'fast' path. Another measure that helped was to enable iparp distributed mode for a more efficient management of IP addresses throughout the BD's blades.

Thank you all for your comments and input on this topic and I hope that our solution helps others that might be experiencing similar issues.