purview receives netflow and mirrorN, but apps are not identified

  • 0
  • 1
  • Problem
  • Updated 2 years ago
  • Solved
Hi,
I installed a virtual vmware purview/analytics AAE in deployment mode 2 (Interface mirrored), with eth0 for netflow and management and eth1 for mirror packet reception:
0. Accept settings and continue
1. Hostname: interpur
2. Deployment Mode: Dual Interface Mirrored
3. Management Interface Configuration (eth0):
Address: 192.168.64.220
Netmask: 255.255.254.0
Gateway: 192.168.64.1
Nameserver: [our dns]
Domain name: [our domain]
4. NIS Server/Domain: Not Configured
5. Monitor Interface Configuration :
Tap Mode Interfaces eth1
An SSA switch sends its netflow packets towards 192.168.64.220 and has a mirror-n towards one of its switch ports which is directly connected to the AAE's host. (AAE's eth1 is mapped to a dedicated vswitch with promiscuous mode and All VLAN (4095) enabled. This vswitch uses the esxi-host's physical adpater which is directly connected to the SSA mirror port ge.1.17.)

SSA mirror and netflowconfig:
# mirror
set mirror create 2
set mirror 2 mirrorN 15
set mirror ports ge.1.17 2
# policy
set policy profile 1 name Purviewmirror pvid-status enable pvid 4095 mirror-destination 2
set policy rule admin-profile port lag.0.4 mask 16 port-string lag.0.4 admin-pid 1
set policy rule admin-profile port lag.0.5 mask 16 port-string lag.0.5 admin-pid 1
set policy rule admin-profile port ge.1.45 mask 16 port-string ge.1.45 admin-pid 1
set policy rule admin-profile port tg.1.4 mask 16 port-string tg.1.4 admin-pid 1
# netflow
set netflow export-interval 1
set netflow export-destination 192.168.64.220 2055
set netflow export-version 9
set netflow port lag.0.4-5 enable rx
set netflow port ge.1.45 enable rx
set netflow port tg.1.4 enable rx
set netflow template refresh-rate 30 timeout 1
set netflow cache enable
AAE receives netflow packets on eth0:
root@interpur:~$ tcpdump -i eth0 udp port 2055
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
11:28:35.047775 IP 192.168.64.9.2055 > interpur.man.htw-berlin.de.2055: UDP, length 1420
11:28:35.052489 IP 192.168.64.9.2055 > interpur.man.htw-berlin.de.2055: UDP, length 1444
11:28:35.058061 IP 192.168.64.9.2055 > interpur.man.htw-berlin.de.2055: UDP, length 1464
...
AAE receives mirror packets on eth1:
root@interpur:~$ tcpdump -i eth1 -c 2
tcpdump: WARNING: eth1: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 65535 bytes
11:32:07.549780 20:b3:99:ad:72:3b (oui Unknown) > 20:b3:99:55:a0:1f (oui Unknown), ethertype Unknown (0x7034), length 70:
0x0000: a001 0800 4500 0034 01b2 4000 3406 395e ....E..4..@.4.9^
0x0010: 40ca 7004 8d2d cdb8 0050 c52b 245a cadb @.p..-...P.+$Z..
0x0020: 12e4 469b 8010 003e e21e 0000 0101 080a ..F....>........
0x0030: 6b12 fa03 36f0 de74 k...6..t
11:32:07.549789 20:b3:99:ad:72:3b (oui Unknown) > 20:b3:99:55:a0:1f (oui Unknown), ethertype Unknown (0x7034), length 64:
0x0000: a001 0800 4500 002c 104b 0000 3006 d058 ....E..,.K..0..X
0x0010: bad4 81d1 8d2d e055 0aac 0913 8d2d e055 .....-.U.....-.U
0x0020: 0000 0000 6002 11b4 5b0f 0000 0204 05ac ....`...[.......
0x0030: 0000 ..

Management Center also shows netflow and mirror packets:


But the Identification Rate stays at 0% and Application Infos are not populated. What could be the reason? AAE was enforced and rebooted.
Photo of htw

htw

  • 1,144 Points 1k badge 2x thumb

Posted 2 years ago

  • 0
  • 1
Photo of Antonio Opromolla

Antonio Opromolla

  • 2,176 Points 2k badge 2x thumb
Hi, do you have enabled jumbo frames on all the path between the sensor and the engine?
(Edited)
Photo of htw

htw

  • 1,144 Points 1k badge 2x thumb
Hi,
on SSA-Switch "show port jumbo ge.1.17" shows jumbo is adminstratively enabled and operational with MTU 10239. 
Photo of Antonio Opromolla

Antonio Opromolla

  • 2,176 Points 2k badge 2x thumb
Ok, ensure that jumbo is enabled along all the path...then you need to wait for a while before the engine collects the flows and display it on the ExtremeControl...
Photo of htw

htw

  • 1,144 Points 1k badge 2x thumb
Jumbo was enabled from the beginning on this switch port, so there is nothing to wait for. The path is quite short becauase the VMs tap-port vswitch is directly connected to the switch port.

edit:
Netsight Console Alarm says: "No bidirectional traffic seen on interface eth1". It seems as if the appliance gets mirror packets, but does not recognize them.
(Edited)
Photo of aloeffle

aloeffle

  • 966 Points 500 badge 2x thumb

Hi.

configure ge.1.17 for 1000 Full Duplex.

set port duplex ge.1.17 full

set port speed ge.1.17 1000

regards

Alex

Photo of Bill Stritzinger

Bill Stritzinger, Alum

  • 6,036 Points 5k badge 2x thumb
Under Analytics, go to configuration, then to location.Make sure the "location" is set with the IP range that you are monitoring.
(Edited)
Photo of htw

htw

  • 1,144 Points 1k badge 2x thumb
Hi,
1000 full duplex was already configured.

edit: Locations exist. (I already used a GRE-tunnel purview successfully with identified apps.)
(Edited)
Photo of Mike Thomas

Mike Thomas, Employee - GTAC - NMS

  • 7,640 Points 5k badge 2x thumb
1. In a 'show port status ge.1.17' the mirrorN port must be operationally and administratively up.
2. The packet capture should be run longer, and make sure that the traffic seen in both directions.

The policy and netflow portions look correct. But lets make sure that that eth1 is receiving bilateral communications from those ports.
Photo of htw

htw

  • 1,144 Points 1k badge 2x thumb
Hi,
>show port status ge.1.17
Port Alias Oper Admin Speed Duplex Type
(truncated) Status Status (bps)
------------ ---------------- -------- ------- ------ ------- ------------------
ge.1.17 up up 1.0G full 1000-t rj45

The mirror setting definitively mirrors bilateral communications. tg.1.4 and ge.1.45 are ports towards external network and lag.0.4-5 are ports to internal networks. These ports do netflow and mirror ingress packets. A GRE-tunnel purview was able to decode bilateral traffic from that n-mirror. I cant paste much longer captures here as there are about 10,000  packets per second.
tcpdump -i eth1 -q
14:45:03.936153 20:b3:99:ad:72:3b (oui Unknown) > 00:16:9c:6e:2f:80 (oui Unknown), Unknown Ethertype (0x7034), length 74:
14:45:03.936299 20:b3:99:ad:72:3b (oui Unknown) > 20:b3:99:55:a0:1f (oui Unknown), Unknown Ethertype (0x7034), length 70:
14:45:03.936420 20:b3:99:ad:72:3b (oui Unknown) > 20:b3:99:55:a0:1f (oui Unknown), Unknown Ethertype (0x7034), length 64:
14:45:03.936425 20:b3:99:ad:72:3b (oui Unknown) > 00:16:9c:6e:2f:80 (oui Unknown), Unknown Ethertype (0x7034), length 77:
14:45:03.936552 20:b3:99:ad:72:3b (oui Unknown) > 20:b3:99:55:a0:1f (oui Unknown), Unknown Ethertype (0x7034), length 99:
14:45:03.936602 20:b3:99:ad:72:3b (oui Unknown) > 00:16:9c:6e:2f:80 (oui Unknown), Unknown Ethertype (0x7034), length 307:
14:45:03.936728 20:b3:99:ad:72:3b (oui Unknown) > 00:16:9c:6e:2f:80 (oui Unknown), Unknown Ethertype (0x7034), length 965:
14:45:03.936850 20:b3:99:ad:72:3b (oui Unknown) > 00:16:9c:6e:2f:80 (oui Unknown), Unknown Ethertype (0x7034), length 171:

I am wondering why the dump from eth1 says "oui unknown" and "ethertype 0x7034" in EVERY packet. This doesnt look normal. Maybe something offsets the packets so AEE cant decode them?
Photo of Mike Thomas

Mike Thomas, Employee - GTAC - NMS

  • 7,640 Points 5k badge 2x thumb
This appears to be one way traffic - 0x7034 appears to be CDP/EDP Discovery protocol packets. So I think your just seeing passive data from attached S-Series port. I dont think data is being mirrored properly from the S-Series.
I would stick a network analyzer on the mirror port directly (ge.1.17) to see if it is forwarding two way traffic, or if the problem is localized possibly to V-Switch.
Photo of htw

htw

  • 1,144 Points 1k badge 2x thumb
Hi,
This is a wireshark trace directly from the n-mirror port:

F.e. we see a paket here which has something to to with SIP. We also see the 0800 ethertype at byte 16 and 17 but it is at the wrong postion. Shouldnt it be at byte 12-13? I dont know where 70 34 and a0 01 do come from. They are in every packet.

Dropboxlink to that pcap-file (about 10 MB): https://www.dropbox.com/s/2tmwfgbyui1tz1k/purviewmirror.pcapng?dl=0

Ethertype 0800 or 8100 (for tagged) is always at byte 16 and 17 in the captures.

SSA-G8018-0652 FW:08.62.01.0034
(Edited)
Photo of Mike Thomas

Mike Thomas, Employee - GTAC - NMS

  • 7,640 Points 5k badge 2x thumb
Where was the trace taken?
Photo of htw

htw

  • 1,144 Points 1k badge 2x thumb
A computer with windows wireshark was directly connected to SSA ge.1.17. 
Photo of Mike Thomas

Mike Thomas, Employee - GTAC - NMS

  • 7,640 Points 5k badge 2x thumb
I would open a GTAC ticket, it does not appear to be an obvious problem.

https://gtacknowledge.extremenetworks.com/articles/How_To/How-to-contact-Extreme-Networks-Global-Tec...
Photo of Jeremy

Jeremy, Embassador

  • 9,788 Points 5k badge 2x thumb
I notice I have these two lines in my config:

   set netflow export-data enable mac
   set netflow export-data enable vlan
Photo of htw

htw

  • 1,144 Points 1k badge 2x thumb
Since it is unlikely that policy mirroring is totally broken I sent the mirror towards another switch port.
set mirror ports ge.1.19 2
Now it worked, packets were clearly decodable. :-)

When I configure the mirror back to ge.1.17 the problem occurs again. So the problem sticks with the switch port. Maybe it is because before trying the policy mirror config, ge.1.17 was used as target port for a GRE-Tunnel before ("tunnel mode gre l2 ge.1.17" in tun-interface definition). The GRE-tunnel mirror worked perfectly. When I started to change to policy mirror configuration I deleted the loopback and tunnel-interfaces. But maybe switch port ge.1.17 still sticks somehow in that mode. I can reboot the router at night to see if ge.1.17 gets "normal" after a reboot.
Photo of Erik Auerswald

Erik Auerswald, Embassador

  • 13,458 Points 10k badge 2x thumb
Since the packets seemed to be encapsulated according to your packet traces, this seems plausible.
Photo of htw

htw

  • 1,144 Points 1k badge 2x thumb
Hi,
after a reboot the affected port returned to normal policy mirror behaviour too.
Photo of Thomas, Frank

Thomas, Frank, Employee

  • 1,902 Points 1k badge 2x thumb
Worth Noting, on the netflow. The reason it's reporting no bi-rdirectional because the netflow config is only sending RX.|

Should look something like "set netflow port ge.1.45 enable both"
Photo of htw

htw

  • 1,144 Points 1k badge 2x thumb
Hi Thomas,
in this case "netflow enable rx" is not a problem. Seeing this SSA as a border router between two networks, ge.1.45 and tg.1.4 are "outside" and the two LAGs are "inside". All of them are set to rx, so traffic getting into the net domain and flowing out of the domain gets detected. When the appliance startet to decodes the traffic properly, the "no bidirectional traffic"-message disappeared.

Purview Deployment guide suggest setting netflow to rx on inside and outside interface f.e. to prevent double netflow detection and since policy mirroring works only ingress too. I wonder if one can combine the ingress mirrors on insinde and outside with a "netflow both" on the inside for Purview detection. hmmm...