topic RE: EXOS Using new feature "Mirroring to Remote IP Addresses" in ExtremeCloud IQ- Site Engine Management Center

EXOS Using new feature "Mirroring to Remote IP Addresses"

M_Nees — Wed, 28 Feb 2018 17:05:00 GMT

How can i use this new 22.4 EXOS feature ?

It sounds perfect to troubleshoot / debug unknown client problems from a remote switch.

I configure this on a EXOS switch in the field and grep the data directly on netsight ubuntu appliance via tcpdump. Unfortunately EXOS encapsulated this traffic into GRE Tunnel.

* Slot-1 12.125.37 # sh mirror
DefaultMirror (Enabled)
Description: Default Mirror Instance, created automatically
Mirror to remote IP: 10.2.0.34 VR : VR-Default
From IP : 10.1.12.125 Ping check: On
Status : Up
Source filter instances used : 1
Port 1:40, all vlans, ingress only

Mirrors defined: 1
Mirrors enabled: 1 (Maximum 4)
HW filter instances used: 1 (Maximum 128)
HW mirror instances used: 1 ingress, 0 egress (Maximum 4 total, 2 egress)
* Slot-1 12.125.38 #09:55:18.931614 IP (tos 0x0, ttl 62, id 0, offset 0, flags [none], proto GRE (47), length 84)
10.1.12.125 > 10.2.0.34: GREv0, Flags [none], length 64
gre-proto-0x88be
09:55:18.931642 IP (tos 0xc0, ttl 64, id 58906, offset 0, flags [none], proto ICMP (1), length 112)
10.2.0.34 > 10.1.12.125: ICMP 10.2.0.34 protocol 47 unreachable, length 92
IP (tos 0x0, ttl 62, id 0, offset 0, flags [none], proto GRE (47), length 84)
10.1.12.125 > 10.2.0.34: GREv0, Flags [none], length 64
gre-proto-0x88be
09:55:18.947159 IP (tos 0x0, ttl 62, id 0, offset 0, flags [none], proto GRE (47), length 84)
10.1.12.125 > 10.2.0.34: GREv0, Flags [none], length 64
gre-proto-0x88be
Is there a (easy) way to look into this GRE Tunnel - prefered with tcpdump ? On Netsight ?

I am looking for a fast troubleshooting way without any graphical GUI and without needing Analytics License - just for analysing packets with tcpdump (but remotely from a EXOS Switch)

Regards,
Matthias

RE: EXOS Using new feature "Mirroring to Remote IP Addresses"

Zdeněk_Pala — Wed, 28 Feb 2018 19:11:00 GMT

if you send the GRE to the Analytics Engine, then the GRE is decapsulated there. probably you can configure GRE on any linux includinging XMC (NetSight). sorry for not giving detailed answer.

RE: EXOS Using new feature "Mirroring to Remote IP Addresses"

M_Nees — Fri, 02 Mar 2018 12:32:00 GMT

Hi James,

i try that on standard XMC like this:

But it seems tcpdump have also a problem (like wireshark) to decode the GRE
tunnel payload:
ip tunnel add sniffer mode gre local 10.0.1.51 remote 10.1.0.4 dev eth0

# ip tunnel
gre0: gre/ip remote any local any ttl inherit nopmtudisc
sniffer: gre/ip remote 10.1.0.4 local 10.0.1.51 dev eth0 ttl inherit

tcpdump -ni sniffer
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on sniffer, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
10:44:15.354690 IP6 fe80::5efe:a00:133 > ff02::2: ICMP6, router solicitation, length 16
10:44:19.362724 IP6 fe80::5efe:a00:133 > ff02::2: ICMP6, router solicitation, length 16
10:45:29.746336 In ethertype Unknown (0x88be), length 258:
0x0000: 68f7 28db 16fc 085b 0e24 eca1 8100 0014 h.(....[.$......
0x0010: 0800 4500 00e0 6077 0000 3811 d4ba 4fe7 ..E...'w..8...O.
0x0020: de4d 0a01 14a6 1194 1194 00cc 0000 1eaa .M..............
0x0030: baac 0000 00ee 7266 478d 1a33 c5a8 815c ......rfG..3...\
...
10:45:29.747665 In ethertype Unknown (0x88be), length 578:
0x0000: 68f7 28db 16fc 085b 0e24 eca1 8100 0014 h.(....[.$......
0x0010: 0800 4500 0220 6079 0000 3811 d378 4fe7 ..E...'y..8..xO.
0x0020: de4d 0a01 14a6 1194 1194 020c 0000 1eaa .M..............
0x0030: baac 0000 00ef e307 8601 82a4 abc7 b80b ................
0x0040: acfd aeaa e360 5207 40c2 1321 1921 4754 .....'R.@..!.!GT
...10:45:29.747849 In ethertype Unknown (0x88be), length 210:
0x0000: 68f7 28db 16fc 085b 0e24 eca1 8100 0014 h.(....[.$......
0x0010: 0800 4500 00b0 607b 0000 3811 d4e6 4fe7 ..E...'{..8...O.
0x0020: de4d 0a01 14a6 1194 1194 009c 0000 1eaa .M..............
...

RE: EXOS Using new feature "Mirroring to Remote IP Addresses"

M_Nees — Fri, 02 Mar 2018 16:11:00 GMT

Hi,

i see that Wireshark natively support to decode GRE tunnel.

So i miror the captured packets directly to my pc.
BUT see the following results:

Wireshark recognize this as (Ciscos) ERSPAN and cannot decapsulate it.

Is there a chance that wireshark can handle this GRE packets ?

If yes this would be much more easier than configuring a GRE Tunnel on XMC or Analytics, just to have the chance to look into mirror packets.

RE: EXOS Using new feature "Mirroring to Remote IP Addresses"

M_Nees — Fri, 02 Mar 2018 16:11:00 GMT

Aruba handle the same issue with this:

Cause : By default, Aruba uses GRE mode 0 which doesn't allow wireshark to decrypt the contents.

Solution : Set the GRE mode to 25944 on both the ends of the tunnel:

https://community.arubanetworks.com/t5/Controller-Based-WLANs/How-can-we-see-the-packets-tunneled-in...

But there must be a change in EXOS Code to modify GRE Mode number ...

RE: EXOS Using new feature "Mirroring to Remote IP Addresses"

Adam_Yedlin — Fri, 02 Mar 2018 16:11:00 GMT

Did you have to change any settings in Wireshark, or did you just point the remote ip to your PC and start a capture?

RE: EXOS Using new feature "Mirroring to Remote IP Addresses"

Stephane_Grosj1 — Wed, 21 Mar 2018 16:44:00 GMT

You should be able to see the GRE traffic from wireshark.

Within wireshark, if you right-click on the ERSPAN header, there is a “protocol preferences” option where you can set “Force to decode fake ERSPAN frame”. This will allow the inner frame to be properly decoded in wireshark.

RE: EXOS Using new feature "Mirroring to Remote IP Addresses"

M_Nees — Thu, 15 Oct 2020 21:28:00 GMT

Working more and more with that feature - very powerfull!

But what about if the packet which needs to be mirrored is at maximum mtu ?

Then the switch is not able to build GRE header around this !

Packets are silently dropped!

That’s very bad for an analysis tool.

Does it help to enable jumbo frames from mirror switch to mirror destination ?

But careless if it would work that’s not practicable in the field.

See here an example:

Classic mirror:

ERSPAN:

big data packets are silently dropped - only ACK are seen

RE: EXOS Using new feature "Mirroring to Remote IP Addresses"

M_Nees — Thu, 15 Oct 2020 21:31:00 GMT

Just a short like what the cisco guys do …

They support truncation:

Configures the MTU size for truncation.
AnySPAN packet that is larger than the configured mtu size is truncated to the configured size with a 4-byte offset.