Switch port packet capture seeing CPU traffic and odd behaviour

[Anonymous]

Hi There,

Been taking some packet captures directly via the command-line on the switch and seeing traffic that I wouldn’t expect. To take one example, I had a port that was experiencing a problem, it was actually negotiating at 10Mb full instead of 1Gb. When I looked at the QoS monitor I could see the QP8 counter incrementing, so thought I would take a look at what that traffic was. Although the result are below I am seeing the same on other ports.

I took a packet capture of the port directly on the switch using the following command:

debug packet capture ports 3:46 on cmd-args "-c 1000" This is an X450G2 stack, running version 22.4.1.4

Here is an example of what I am seeing below:

10.65.232.250 is the management IP address of the switch itself.

10.65.232.252 is the physical address of the Management VLAN (just for this stack) on one of the cores for this switch.

10.65.80.252 is the physical address of the Voice VLAN on one of the cores for this switch.

So the network seems to be running fine, CPU on cores is about 2.5% and the edges about 5%. When I look at the packet captures of the ports and CPU’s I’m seeing odd things and trying to understand whats going on.

Take the example below, this is from the same port capture just filtered for VRRP. I’m seeing VRRP hellos every second as you would expect, but I'm seeing them 3 at a time?

I’m also seeing VRRP hellos for the Voice VLAN (10.65.80.252) that isn’t tagged on this port but being sent perhaps 30 times in one second?

So I’m unsure 1st why I am seeing this traffic on a port capture and why I’m seeing the traffic I am? Looks like a problem, possible loop but can't find the source. I'm seem to be getting the same results in other places but nothing is common, i.e. on a different switch I see the same thing but for the VLANs on that switch?

Here is an overview of the topology:

The summary of the network is that there are 2 different buildings A & B, each have a pair of cores MLAG'ed together. There is a VRID per site for VLANs in that building only and a common VRID for the VLANs that are common on each core. This is configured with fabric routing and OSPF. The stack in question would be one in the top right.

The switches at the top are legacy Cisco switches, they are currently connected using VRRP VRID 101 (VLAN 999).



Just wondering if anyone had any suggestions?

[OscarK]

Hello Martin, the CPU capture will capture all from and to CPU, only when trying to capture on a port be aware all traffic is copied to the CPU and might overwhelm the CPU is you mirror a loaded 10 Gig port.
port mirror and remote mirroring are the methods to take a reliable port mirror.

[Anonymous]

Thanks Oscar

[Anonymous]

Thanks for posting back Oscar, much appreciated....

Wow, OK. Well that's good and bad news to know as I was going crazy trying to interpret what I was seeing in the packet captures... especially those VRRP hellos being sent from another VLAN 30 times in 1 second!?

Assume the same can happen on a CPU capture, as seeing a similar thing there?

So in order to get a packet capture I that is true to whats actually on the wire or the CPU... CPU aside for now, is it best to do a port mirror? Can I use remote mirroring, think I see a new command in version 22.4.x where I can send it to an IP?

Not sure what you would do about the CPU?

Its immeasurably useful to be able to do packet captures directly on switch so I can do things remotely, and generally this is still OK but be useful nonetheless to trust whats in them so that I'm not chasing my tail. So anything you might know to assist where this can be enhanced remotely or onsite that would be additionally really useful.

Perhaps the best way is with just an inline hub?

Many thanks in advance

[OscarK]

When doing packet capture through debug packet you will see packets multiple times, before and after the tag is added, sometimes you see incoming traffic and outgoing traffic from the CPU perspective. When you take a capture from a port that traffic is send to the CPU which then captures the packets. This means you will see CPU traffic and the port traffic.
Allthough debug packet works in most cases it is still a debug command which is not tested a lot and hence results can vary.