topic RE: purview receives netflow and mirrorN, but apps are not identified in ExtremeCloud IQ- Site Engine Management Center

purview receives netflow and mirrorN, but apps are not identified

htw — Tue, 08 Nov 2016 17:42:00 GMT

Hi,
I installed a virtual vmware purview/analytics AAE in deployment mode 2 (Interface mirrored), with eth0 for netflow and management and eth1 for mirror packet reception:
0. Accept settings and continue
1. Hostname: interpur
2. Deployment Mode: Dual Interface Mirrored
3. Management Interface Configuration (eth0):
Address: 192.168.64.220
Netmask: 255.255.254.0
Gateway: 192.168.64.1
Nameserver: [our dns]
Domain name: [our domain]
4. NIS Server/Domain: Not Configured
5. Monitor Interface Configuration :
Tap Mode Interfaces eth1
An SSA switch sends its netflow packets towards 192.168.64.220 and has a mirror-n towards one of its switch ports which is directly connected to the AAE's host. (AAE's eth1 is mapped to a dedicated vswitch with promiscuous mode and All VLAN (4095) enabled. This vswitch uses the esxi-host's physical adpater which is directly connected to the SSA mirror port ge.1.17.)

SSA mirror and netflowconfig:
# mirror
set mirror create 2
set mirror 2 mirrorN 15
set mirror ports ge.1.17 2
# policy
set policy profile 1 name Purviewmirror pvid-status enable pvid 4095 mirror-destination 2
set policy rule admin-profile port lag.0.4 mask 16 port-string lag.0.4 admin-pid 1
set policy rule admin-profile port lag.0.5 mask 16 port-string lag.0.5 admin-pid 1
set policy rule admin-profile port ge.1.45 mask 16 port-string ge.1.45 admin-pid 1
set policy rule admin-profile port tg.1.4 mask 16 port-string tg.1.4 admin-pid 1
# netflow
set netflow export-interval 1
set netflow export-destination 192.168.64.220 2055
set netflow export-version 9
set netflow port lag.0.4-5 enable rx
set netflow port ge.1.45 enable rx
set netflow port tg.1.4 enable rx
set netflow template refresh-rate 30 timeout 1
set netflow cache enableAAE receives netflow packets on eth0:
root@interpur:~$ tcpdump -i eth0 udp port 2055
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
11:28:35.047775 IP 192.168.64.9.2055 > interpur.man.htw-berlin.de.2055: UDP, length 1420
11:28:35.052489 IP 192.168.64.9.2055 > interpur.man.htw-berlin.de.2055: UDP, length 1444
11:28:35.058061 IP 192.168.64.9.2055 > interpur.man.htw-berlin.de.2055: UDP, length 1464
...AAE receives mirror packets on eth1:
root@interpur:~$ tcpdump -i eth1 -c 2
tcpdump: WARNING: eth1: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 65535 bytes
11:32:07.549780 20:b3:99??72:3b (oui Unknown) > 20:b3:99:55:a0:1f (oui Unknown), ethertype Unknown (0x7034), length 70:
0x0000: a001 0800 4500 0034 01b2 4000 3406 395e ....E..4..@.4.9^
0x0010: 40ca 7004 8d2d cdb8 0050 c52b 245a cadb @.p..-...P.+$Z..
0x0020: 12e4 469b 8010 003e e21e 0000 0101 080a ..F....>........
0x0030: 6b12 fa03 36f0 de74 k...6..t
11:32:07.549789 20:b3:99??72:3b (oui Unknown) > 20:b3:99:55:a0:1f (oui Unknown), ethertype Unknown (0x7034), length 64:
0x0000: a001 0800 4500 002c 104b 0000 3006 d058 ....E..,.K..0..X
0x0010: bad4 81d1 8d2d e055 0aac 0913 8d2d e055 .....-.U.....-.U
0x0020: 0000 0000 6002 11b4 5b0f 0000 0204 05ac ....'...[.......
0x0030: 0000 ..
Management Center also shows netflow and mirror packets:

But the Identification Rate stays at 0% and Application Infos are not populated. What could be the reason? AAE was enforced and rebooted.

RE: purview receives netflow and mirrorN, but apps are not identified

Antonio_Opromol — Tue, 08 Nov 2016 18:01:00 GMT

Hi, do you have enabled jumbo frames on all the path between the sensor and the engine?

RE: purview receives netflow and mirrorN, but apps are not identified

htw — Tue, 08 Nov 2016 18:30:00 GMT

Hi,
on SSA-Switch "show port jumbo ge.1.17" shows jumbo is adminstratively enabled and operational with MTU 10239.

RE: purview receives netflow and mirrorN, but apps are not identified

Antonio_Opromol — Tue, 08 Nov 2016 18:33:00 GMT

Ok, ensure that jumbo is enabled along all the path...then you need to wait for a while before the engine collects the flows and display it on the ExtremeControl...

RE: purview receives netflow and mirrorN, but apps are not identified

htw — Tue, 08 Nov 2016 18:40:00 GMT

Jumbo was enabled from the beginning on this switch port, so there is nothing to wait for. The path is quite short becauase the VMs tap-port vswitch is directly connected to the switch port.

edit:
Netsight Console Alarm says: "No bidirectional traffic seen on interface eth1". It seems as if the appliance gets mirror packets, but does not recognize them.

RE: purview receives netflow and mirrorN, but apps are not identified

aloeffle — Tue, 08 Nov 2016 19:10:00 GMT

Hi.

configure ge.1.17 for 1000 Full Duplex.

set port duplex ge.1.17 full

set port speed ge.1.17 1000

regards

Alex

RE: purview receives netflow and mirrorN, but apps are not identified

Bill_Stritzinge — Tue, 08 Nov 2016 19:18:00 GMT

Under Analytics, go to configuration, then to location.Make sure the "location" is set with the IP range that you are monitoring.

RE: purview receives netflow and mirrorN, but apps are not identified

htw — Tue, 08 Nov 2016 19:29:00 GMT

Hi,
1000 full duplex was already configured.

edit: Locations exist. (I already used a GRE-tunnel purview successfully with identified apps.)

RE: purview receives netflow and mirrorN, but apps are not identified

Mike_Thomas — Tue, 08 Nov 2016 20:24:00 GMT

1. In a 'show port status ge.1.17' the mirrorN port must be operationally and administratively up.
2. The packet capture should be run longer, and make sure that the traffic seen in both directions.

The policy and netflow portions look correct. But lets make sure that that eth1 is receiving bilateral communications from those ports.

RE: purview receives netflow and mirrorN, but apps are not identified

htw — Tue, 08 Nov 2016 20:51:00 GMT

Hi,
>show port status ge.1.17
Port Alias Oper Admin Speed Duplex Type
(truncated) Status Status (bps)
------------ ---------------- -------- ------- ------ ------- ------------------
ge.1.17 up up 1.0G full 1000-t rj45

The mirror setting definitively mirrors bilateral communications. tg.1.4 and ge.1.45 are ports towards external network and lag.0.4-5 are ports to internal networks. These ports do netflow and mirror ingress packets. A GRE-tunnel purview was able to decode bilateral traffic from that n-mirror. I cant paste much longer captures here as there are about 10,000 packets per second.
tcpdump -i eth1 -q
14:45:03.936153 20:b3:99??72:3b (oui Unknown) > 00:16:9c:6e:2f:80 (oui Unknown), Unknown Ethertype (0x7034), length 74:
14:45:03.936299 20:b3:99??72:3b (oui Unknown) > 20:b3:99:55:a0:1f (oui Unknown), Unknown Ethertype (0x7034), length 70:
14:45:03.936420 20:b3:99??72:3b (oui Unknown) > 20:b3:99:55:a0:1f (oui Unknown), Unknown Ethertype (0x7034), length 64:
14:45:03.936425 20:b3:99??72:3b (oui Unknown) > 00:16:9c:6e:2f:80 (oui Unknown), Unknown Ethertype (0x7034), length 77:
14:45:03.936552 20:b3:99??72:3b (oui Unknown) > 20:b3:99:55:a0:1f (oui Unknown), Unknown Ethertype (0x7034), length 99:
14:45:03.936602 20:b3:99??72:3b (oui Unknown) > 00:16:9c:6e:2f:80 (oui Unknown), Unknown Ethertype (0x7034), length 307:
14:45:03.936728 20:b3:99??72:3b (oui Unknown) > 00:16:9c:6e:2f:80 (oui Unknown), Unknown Ethertype (0x7034), length 965:
14:45:03.936850 20:b3:99??72:3b (oui Unknown) > 00:16:9c:6e:2f:80 (oui Unknown), Unknown Ethertype (0x7034), length 171:

I am wondering why the dump from eth1 says "oui unknown" and "ethertype 0x7034" in EVERY packet. This doesnt look normal. Maybe something offsets the packets so AEE cant decode them?

RE: purview receives netflow and mirrorN, but apps are not identified

Mike_Thomas — Tue, 08 Nov 2016 21:23:00 GMT

This appears to be one way traffic - 0x7034 appears to be CDP/EDP Discovery protocol packets. So I think your just seeing passive data from attached S-Series port. I dont think data is being mirrored properly from the S-Series.
I would stick a network analyzer on the mirror port directly (ge.1.17) to see if it is forwarding two way traffic, or if the problem is localized possibly to V-Switch.

RE: purview receives netflow and mirrorN, but apps are not identified

htw — Tue, 08 Nov 2016 23:20:00 GMT

Hi,
This is a wireshark trace directly from the n-mirror port:

F.e. we see a paket here which has something to to with SIP. We also see the 0800 ethertype at byte 16 and 17 but it is at the wrong postion. Shouldnt it be at byte 12-13? I dont know where 70 34 and a0 01 do come from. They are in every packet.

Dropboxlink to that pcap-file (about 10 MB): https://www.dropbox.com/s/2tmwfgbyui1tz1k/purviewmirror.pc.png?dl=0

Ethertype 0800 or 8100 (for tagged) is always at byte 16 and 17 in the captures.

SSA-G8018-0652 FW:08.62.01.0034

RE: purview receives netflow and mirrorN, but apps are not identified

Mike_Thomas — Wed, 09 Nov 2016 01:44:00 GMT

Where was the trace taken?

RE: purview receives netflow and mirrorN, but apps are not identified

htw — Wed, 09 Nov 2016 03:30:00 GMT

A computer with windows wireshark was directly connected to SSA ge.1.17.

RE: purview receives netflow and mirrorN, but apps are not identified

Mike_Thomas — Wed, 09 Nov 2016 03:37:00 GMT

I would open a GTAC ticket, it does not appear to be an obvious problem.

https://gtacknowledge.extremenetworks.com/articles/How_To/How-to-contact-Extreme-Networks-Global-Tec...

RE: purview receives netflow and mirrorN, but apps are not identified

Jeremy_Gibbs — Wed, 09 Nov 2016 13:11:00 GMT

I notice I have these two lines in my config:

set netflow export-data enable mac
set netflow export-data enable vlan

RE: purview receives netflow and mirrorN, but apps are not identified

htw — Wed, 09 Nov 2016 17:24:00 GMT

Since it is unlikely that policy mirroring is totally broken I sent the mirror towards another switch port.
set mirror ports ge.1.19 2Now it worked, packets were clearly decodable. 🙂

When I configure the mirror back to ge.1.17 the problem occurs again. So the problem sticks with the switch port. Maybe it is because before trying the policy mirror config, ge.1.17 was used as target port for a GRE-Tunnel before ("tunnel mode gre l2 ge.1.17" in tun-interface definition). The GRE-tunnel mirror worked perfectly. When I started to change to policy mirror configuration I deleted the loopback and tunnel-interfaces. But maybe switch port ge.1.17 still sticks somehow in that mode. I can reboot the router at night to see if ge.1.17 gets "normal" after a reboot.

RE: purview receives netflow and mirrorN, but apps are not identified

Erik_Auerswald — Wed, 09 Nov 2016 17:24:00 GMT

Since the packets seemed to be encapsulated according to your packet traces, this seems plausible.

RE: purview receives netflow and mirrorN, but apps are not identified

htw — Thu, 10 Nov 2016 14:37:00 GMT

Hi,
after a reboot the affected port returned to normal policy mirror behaviour too.

RE: purview receives netflow and mirrorN, but apps are not identified

Frank_Thomas — Thu, 10 Nov 2016 23:05:00 GMT

Worth Noting, on the netflow. The reason it's reporting no bi-rdirectional because the netflow config is only sending RX.|

Should look something like "set netflow port ge.1.45 enable both"