topic RE: Not all packets are forwarded in ExtremeSwitching (EXOS/Switch Engine)

Not all packets are forwarded

Leon_Wessels — Mon, 18 Apr 2016 20:19:00 GMT

When I enable port mirroring on our Extreme Summit x460-24t not all the packets are sent to the mirror destination port. In a wireshark capture from the mirror port I only see very few (less than 1% of the actual) audio packets (ieee1722.cdfield == 0). When doing a packet capture using another (non-extreme) switch I see all the packets.

To enable port mirroring I used these commands:

# enable mirror lavb to port 1
# config mirror lavb add port 9-15
# show mirror

DefaultMirror (Disabled)
Description: Default Mirror Instance, created automatically
Mirror to port: 1
Source filters configured :
Ports 9-24, all vlans, ingress and egress

lavb (Enabled)
Description:
Mirror to port: 1
Source filter instances used : 7
Port 9, all vlans, ingress and egress
Port 10, all vlans, ingress and egress
Port 11, all vlans, ingress and egress
Port 12, all vlans, ingress and egress
Port 13, all vlans, ingress and egress
Port 14, all vlans, ingress and egress
Port 15, all vlans, ingress and egress

Mirrors defined: 2
Mirrors enabled: 1 (Maximum 4)
HW filter instances used: 7 (Maximum 128)
HW mirror instances used: 1 ingress, 1 egress (Maximum 4 total, 2 egress)

Switch version:
* X460-24p.2 # show version
Switch : 800323-00-05 1120G-81240 Rev 5.0 BootROM: 2.0.1.7 IMG: 16.1.1.4
PSU-1 : PSSF751301A-11 800382-00-04 1120A-40142
PSU-2 :

Image : ExtremeXOS version 16.1.1.4 16.1.1.4-patch1-1 by release-manager
on Thu Aug 6 16:14:58 EDT 2015
BootROM : 2.0.1.7
Diagnostics : 6.3

Am I missing something in the configuration? Each of those ports has a maximum of 18Mbps traffic so the total traffic should be much less than the 1Gbps allowed by the port being mirrored to.

RE: Not all packets are forwarded

Bill_Stritzinge — Mon, 18 Apr 2016 22:52:00 GMT

Leon,

Your configuration looks OK. Have you checked the drivers of the NIC on the laptop or machine running wireshark? I have seen where issues on the drivers prevents sometimes packets with tags and other strange things and/or traffic not being seen. Check that and let us know...

Bill

RE: Not all packets are forwarded

Leon_Wessels — Mon, 18 Apr 2016 22:52:00 GMT

Hi

I'm using Ubuntu 14.04 with kernel 3.13.0-85-generic.
I tried these three network cards (output from lspci)
01:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM57762 Gigabit Ethernet PCIe
03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06)
06:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)

I also tried the following on the pc:
# sudo vconfig add eth2 2
# sudo cat /proc/net/vlan/config
VLAN Dev name | VLAN ID
Name-Type: VLAN_NAME_TYPE_RAW_PLUS_VID_NO_PAD
eth2.2 | 2 | eth2
# sudo cat /proc/net/vlan/eth2.2
eth2.2 VID: 2 REORDER_HDR: 1 dev->priv_flags: 1
total frames received 1166
total bytes received 261184
Broadcast/Multicast Rcvd 1166

total frames transmitted 0
total bytes transmitted 0
Device: eth2
INGRESS priority mappings: 0:0 1:0 2:0 3:0 4:0 5:0 6:0 7:0
EGRESS priority mappings:

On the switch it seems that the avb packets have vlan tag 2.
# show vlan SYS_VLAN_0002
VLAN Interface with name SYS_VLAN_0002 created dynamically
Admin State: Enabled Tagging: 802.1Q Tag 2
Description: None
Virtual router: VR-Default
IPv4 Forwarding: Disabled
IPv4 MC Forwarding: Disabled
IPv6 Forwarding: Disabled
IPv6 MC Forwarding: Disabled
IPv6: None
STPD: s0(Disabled)
Protocol: Match all unfiltered protocols
Loopback: Disabled
NetLogin: Disabled
OpenFlow: Disabled
TRILL: Disabled
QosProfile: None configured
Flood Rate Limit QosProfile: None configured
Ports: 15. (Number of active ports=15)
Tag: *5H, *9H, *11H, *13H, *14H, *16H, *17H,
*20H, *21H, *22H, *23H, *24H

None of this made any difference. Is there any way to forward the packets without the vlan tag?

This command:
# enable mirror lavb to port 1

Produces this output:
WARNING: This command will remove VLAN membership from the monitor port.
Do you want to continue? (y/N) Yes

Should I be concerned about it at all?

Regards
Leon

RE: Not all packets are forwarded

Derek_Bird — Tue, 19 Apr 2016 03:22:00 GMT

Leon:

It's important that promiscuous mode is enabled on the interface you're capturing with. Here's what you're looking for in capture options in Wireshark:

RE: Not all packets are forwarded

Leon_Wessels — Tue, 19 Apr 2016 03:22:00 GMT

I checked and promiscuous mode was enabled.

RE: Not all packets are forwarded

Anonymous — Tue, 19 Apr 2016 14:31:00 GMT

Perhaps do a 'show ports congestion', see if any packets are being dropped?

RE: Not all packets are forwarded

Leon_Wessels — Tue, 19 Apr 2016 14:31:00 GMT

Thanks for the tip. This seems to be the problem. About 7000 packets are dropped per second. I changed my mirror configuration so that I mirror a single port and the number of packets dropped per second seems to be about the same. I don't know how to fix this though. I don't see any reason why the packets should be dropped.

Output that might be useful to debug the dropped packets:
# show port 1 buffer
Packet Buffer Allocation for ports in range 1-30
Total Packet Buffer Size: 3145728 bytes, Not Overcommitted
Total Shared Buffer Size: 2850304
Port 1 Max Shared Buffer Usage: 569984 bytes (20%)
QP1: Reserved Buffer: 1536 bytes
QP6: Reserved Buffer: 1536 bytes
QP7: Reserved Buffer: 1536 bytes
QP8: Reserved Buffer: 1536 bytes

# show port 1 configuration
Port Configuration Monitor Tue Apr 19 18:00:04 2016
Port Virtual Port Link Auto Speed Duplex Flow Load Media
router State State Neg Cfg Actual Cfg Actual Cntrl Master Pri Red
================================================================================
1 VR-Default E A ON AUTO 1000 AUTO FULL SY/ASY UTP

# show port 1 information
Port Flags Link ELSM Link Num Num Num Jumbo QOS Load
State /OAM UPS STP VLAN Proto Size profile Master
=====================================================================================
1 Em------e--fMB----- active - / - 0 0 0 0 9216 none
=====================================================================================
> indicates Port Display Name truncated past 8 characters
Flags : a - Load Sharing Algorithm address-based,
b - Rx and Tx Flow Control Enabled, B - Broadcast Flooding Enabled,
D - Port Disabled, e - Extreme Discovery Protocol Enabled,
E - Port Enabled, f - Unicast Flooding Enabled,
F - Priority Flow Control Enabled, G - MLAG Enabled, i - Isolation,
j - Jumbo Frame Enabled, l - Load Sharing Enabled,
L - Extreme Link Status Monitoring Enabled,
m - MACLearning Enabled, M - Multicast Flooding Enabled,
n - Ingress TOS Enabled, o - Dot1p Replacement Enabled,
O - Ethernet OAM Enabled, p - Load Sharing Algorithm port-based,
P - Software redundant port(Primary),
R - Software redundant port(Redundant), s - diffserv Replacement Enabled,
v - Vman Enabled, w - MACLearning Disabled with Forwarding,
x - Rx Flow Control Enabled

//the counters of both ports were reset at the same time.
# show port 1,9 packet
Port Packet Statistics Tue Apr 19 18:05:05 2016
Port Link Packet Sizes
State 0-64 65-127 128-255 256-511 512-1023 1024-1518 Jumbo
================================================================================
1 A 8 198 79 1 0 0 0
9 A 71 121 32346 1 0 0 0

For reference this is the new mirror setup:
#show mirror

DefaultMirror (Disabled)
Description: Default Mirror Instance, created automatically
Mirror to port: -

lavb (Enabled)
Description:
Mirror to port: 1
Source filter instances used : 1
Port 9, all vlans, ingress and egress

Mirrors defined: 2
Mirrors enabled: 1 (Maximum 4)
HW filter instances used: 1 (Maximum 128)
HW mirror instances used: 1 ingress, 1 egress (Maximum 4 total, 2 egress)

RE: Not all packets are forwarded

Leon_Wessels — Tue, 19 Apr 2016 15:26:00 GMT

Thanks to all for the help. The dropped packets seem to be caused when avb is enabled on the port being mirrored to. Disabling avb on that port before enabling the mirroring fixes this issue.

For future reference have a look at this article:
https://gtacknowledge.extremenetworks.com/articles/Solution/AVB-Mirroring-AVB-Control-Protocol-Traff...