NetFlow showing impossible flows

  • 0
  • 1
  • Problem
  • Updated 2 years ago
  • Solved
Good morning, everyone.  Last week, I configured one of our FortiGate 100D firewalls to send NetFlow datagrams to my new Management Center server.  I began seeing data immediately.  However, I'm receiving reports of impossible flows:



I sorted the flows in descending order of TX and RX Bytes in those two images, respectively.

It's just not a possible amount of traffic.  This isn't the first time I've encountered these crazy NetFlow statistics with Management Center, though.  Last year, at a different organization, I was testing the feasibility of using Management Center to manage a Cisco network (it isn't very feasible, by the way), and I encountered similar impossible flows when I enabled reporting our our Cisco Catalyst 6513.

My evaluation license in Management Center expired before I discovered the source of those impossible flows.

The situation at my organization now is that we have fielded Summit X440 switches at remote locations, which are using FortiGate 100D firewalls as their gateways.  From a flow monitoring perspective of network management, it makes sense to collect the flow data from the firewalls (which are capable of NetFlow v9, required by Management Center).  The X440 switches are not able to report Netflow, and Management Center is unable to collect sFlow (which the X440 switches can report).

I've asked the Fortinet community about the possibility of filtering the NetFlow datagrams at the source, although their user community seems to be largely a ghost town.

Is it possible for me to ignore these flows at the Management Center?  They are making half of the Analytics dashboard useless, by filling panes that sort by bandwidth with noise:



Any dashboard pane that reports TopN of anything by bandwidth shows those impossible flows, masking anything of use to me about our networks.

How to make Management Center report on flows only concerning our networks?  Or, how to make it ignore that noise?  Those are the problems I need to solve to make NetSight Analytics of any use to us.
Photo of Jesse Ohlsson

Jesse Ohlsson

  • 680 Points 500 badge 2x thumb

Posted 2 years ago

  • 0
  • 1
Photo of Jeremy

Jeremy, Embassador

  • 9,788 Points 5k badge 2x thumb
If NETFLOW is reporting it, I bet it's actually true and you have more of a problem than you think.  
Photo of Jeremy

Jeremy, Embassador

  • 9,788 Points 5k badge 2x thumb
Also, my brain instantly screamed routing loop when I saw that.
Photo of Jesse Ohlsson

Jesse Ohlsson

  • 680 Points 500 badge 2x thumb
You'd think, Jeremey.  But, I'm not routing any of the source or destination addresses of those crazy flows.  And, I have no latency or throughput issues with these remote sites.  That detracts from the routing loop possibility.

What puzzles me is that I saw this same sort of crazy stuff from that other Cisco router I once configured to send NetFlow to Management Center.

This can't possibly be traffic through my monitored interface.  The thing I can't get my mind around is from where is it coming?
Photo of Jeremy

Jeremy, Embassador

  • 9,788 Points 5k badge 2x thumb
Try doing some packet captures at the ingress of your network (where the firewall is).  Could you make sure you have Null routes black holing everything else that you don't route for? 
Photo of Jesse Ohlsson

Jesse Ohlsson

  • 680 Points 500 badge 2x thumb
On page 853 of the Extreme Management Center User Guide, is a section called Flow Collector Filter:

Use this field to filter all incoming flows as they are processed by the flow
collector. Flows not matching the filter are discarded and not maintained in
memory on the server. If you add a filter here, the current flows stored in the
cache are trimmed to only matching flows.
Use this option if you want to use flow collection to look for specific results,
and unrelated flows do not need to be processed. For example, only
processing flows pertaining to a particular subnet."
That looks promising, but there's nothing at all in that manual describing the syntax to construct that filter.
Photo of Jesse Ohlsson

Jesse Ohlsson

  • 680 Points 500 badge 2x thumb
I'll investigate those null routes, Jeremy.
Photo of Jeremy

Jeremy, Embassador

  • 9,788 Points 5k badge 2x thumb
This is what I use on our edge route.

ip route 10.0.0.0 255.0.0.0 Null0 200

ip route 169.254.0.0 255.255.0.0 Null0

ip route 192.168.0.0 255.255.0.0 Null0 200

ip route 172.16.0.0 255.240.0.0 Null0


You may need to adjust for your network.

Photo of Jesse Ohlsson

Jesse Ohlsson

  • 680 Points 500 badge 2x thumb
Jeremy, the clients and servers in those impossibly large flows aren't in any of my networks.  What I mean to say here, I think, is that it isn't one of the non-routable networks as you've listed that are the sources of these flows I'm investigating.
Photo of EtherMAN

EtherMAN, Embassador

  • 6,688 Points 5k badge 2x thumb
Before we purchased Solarwinds we used this free sflow collector ... works great just does not create a data base for searches but for top talkers and such it is good... Vhttp://www.inmon.com/products/sFlowTrend.php  .. This would give you the layer 2 traffic on the extreme side to compare to what you are getting on the firewall via netflow. 
Photo of Jesse Ohlsson

Jesse Ohlsson

  • 680 Points 500 badge 2x thumb
I'm collecting sFlow from the Extreme switches using PRTG as the collector.  That output patches well what the NetFlow from the firewall reports, if I ignore the impossible flows I mentioned earlier.
Photo of Jesse Ohlsson

Jesse Ohlsson

  • 680 Points 500 badge 2x thumb
On the Management Center > Analytics page, there exists a search/filter field, with which to operate on the Flows tab of that page.  Contextual help shows a syntax for developing filters for that page:

Enter search criteria to filter table. One or more filters may be entered, separated by semi-colons. For each filter, individual components of that filter are comma separated. Specific flow components must be specified with a key.

Keys are not case sensitive.


Supported keys: CLIENT, SERVER, SERVERPORT, PROTO, SENSOR, IF, INIF, OUTIF, APP, APPGROUP, META, LOCATION, DEVFAMILY, USER, PROFILE
*Note: CLIENT/SERVER may be an IP (CIDR Mask Supported) or host, and may contain a port (X.X.X.X:P)
Supported modifiers: duration, size, packets, bps, with a comparator of: >, >=, =, <, or <=
Supported keywords: Include/Exclude (Include is implied)
Fuzzy filters and partial hostnames are supported
    *ie "Doe" will return flows w/ hostnames "JonDoe", "JillDoe", etc


Additionally flow metadata can be searched by using a colon as the key-value delimiter.
Example: meta=Content-Type:application/json


Everything is case insensitive.
A Fuzzy filter is a string without a keyword. It can partially match any value in any column. E.g. "Google" would match source or destination host name "www.google.com";, or application names "Google Mail" or "Google News".


Examples (single filter):
1) "client=JonDoe, server=10.20.77.33, snmp" –> SNMP Flows from host JonDoe to 10.20.77.33
2) "JonDoe, SENSOR=10.20.77.33, snmp" –> SNMP Flows from netflow sensor 10.20.77.33 To/From JonDoe
3) "SERVERPORT=161, duration>1000, exclude" –> All flows EXCEPT SNMP(161) flows that lasted more than 1 second
4) "snmp" –> All snmp traffic
5) "10.20.77.33:161" –> flows where server/serverport = 10.20.77.33:161
6) "JonDoe:161" –> snmp flows to or from JonDoe
7) app=DNS –> All flows identified as belonging to the application "DNS"
8) app=89 –> All flows identified as belonging to the application whose ID is 89
9) appgroup=Social Networking –> All flows identified as belonging to the application group "Social Networking"
10) user=none,exclude –> All flows with an identified user
11) inif=11002 –> All flows where input interface = 11002
12) meta=snmp–> All flows with metadata containing the text SNMP
13) meta=Content-Type:application/json;> All flows with an application type of JSON


Example (multi filter):
1) "SERVERPORT=161, duration>1000, exclude; CLIENT=JonDoe; CLIENT=JillDoe" –> All flows from JonDoe and JillDoe except SNMP flows lasting more than 1 sec

If the filter fails to find a match, no data is displayed

Does anyone know if this filter syntax can be used in the Administration > Options > NetFlow Collection > Advanced > Flow Collector Filter field?
Photo of Jeremy

Jeremy, Embassador

  • 9,788 Points 5k badge 2x thumb
Photo of Jeremy

Jeremy, Embassador

  • 9,788 Points 5k badge 2x thumb
Better question, do you have reverse path check enabled on your FG?
Photo of Jesse Ohlsson

Jesse Ohlsson

  • 680 Points 500 badge 2x thumb
Jeremy, I learned that Fortinet calls it reverse path forwarding, and it is enabled by default.  There exists a command to permit asymmetric routing (set asymroute enable), which is not enabled on this firewall.

This mystery continues.
Photo of Pala, Zdenek

Pala, Zdenek, Employee

  • 9,160 Points 5k badge 2x thumb
I met following behavior with Fortigate:
the Fortigate does cache netflow records. if the netflow destination is unreachable all records are cached. when the netflow destination is available again, you can get aggregated data.
can that explain your observation?
Photo of Jesse Ohlsson

Jesse Ohlsson

  • 680 Points 500 badge 2x thumb
I wouldn't think so.  Since I enabled Netflow on that fortigate, the collector (Extreme Management Center) has been available.

Even if not, those flows are describing volumes of traffic that wouldn't be possible for thousands of years over the size pipe to which the fortigate is connected.  I really think it's a fault in how the Fortigate reported it, but I don't know how to either filter it out, or delete the information about those flows in Management Center.

My post on the Fortinet community remains unread and unanswered.

Here's a peculiar  thing, though:  The NetFlow sensor on our PRTG server never once reported these fantastic flows from our Fortigate firewalls.

With what I'm struggling is where lies the fault?  Is the Fortigate reporting garbage data?  If so, I can't figure why PRTG didn't display it, or, why Management Center analytics does display it.

I strongly suspect that Management Center is displaying what's been reported to it, and PRTG is filtering the NetFlow data before displaying it.  Just a suspicion, though.  I can't figure how PRTG would know what to filter out, and I didn't tell it to do so.
Photo of Erik Auerswald

Erik Auerswald, Embassador

  • 12,960 Points 10k badge 2x thumb
Hi Jesse,

did you try to take a packet capture of the Netflow data sent by the Fortigate, and looked at it using Wireshark?

The definitely impossible flows show neither input nor output interfaces. They do show some regularity. That might hint at some packet format not supported by Extreme Management Center and interpreted incorrectly.

HTH,
Erik
Photo of Jesse Ohlsson

Jesse Ohlsson

  • 680 Points 500 badge 2x thumb
Hello, Erik.

I took your advice and captured 10000 NetFlow packets, which bracketed (in time) instances of impossible flows as shown again on the Analytics Dashboard.

None of the source IP addresses, destination IP addresses, or traffic protocols of the impossible flows displayed by the dashboard during the time period of my packet capture exist in the capture file.

I'm getting fairly good at using WireShark, looking for this stuff.

I think it's time to open a case with GTAC to investigate further.
Photo of Erik Auerswald

Erik Auerswald, Embassador

  • 12,960 Points 10k badge 2x thumb
Hello Jesse,

I suspect that some of the NetFlow packets use a format not supported by Extreme Management Center, but instead of dropping or ignoring them, EMC interprets them as the impossible flows you see.

Asking GTAC to take a look at this seems like a good idea. :-)

Erik
Photo of Jesse Ohlsson

Jesse Ohlsson

  • 680 Points 500 badge 2x thumb
Still having this problem.  Can anyone point me to the syntax to use in the Administration > Options > NetFlow Collection > Advanced > Flow Collector Filter field?

I did a packet capture of all exports over a period of several hours, which bracketed some of the wildly impossible reports displayed in the Analytics Dashboard.  No instances of any identifying piece of the offending dashboard graphs existed in that packet capture file.

This lends support to the suspicion that it is Extreme Management Center that is making the error.

However, if I can find out how to complete that filtering field, I can pare the NetFlow exports down to only those that describe traffic to or from my specific IP networks.
Photo of Ronald Dvorak

Ronald Dvorak, Embassador

  • 47,526 Points 20k badge 2x thumb
Hover over the search field with the mouse pointer till the info field show and click on "more" to get the options and examples....