Header Only - DO NOT REMOVE - Extreme Networks

NetFlow showing impossible flows


Userlevel 1
Good morning, everyone. Last week, I configured one of our FortiGate 100D firewalls to send NetFlow datagrams to my new Management Center server. I began seeing data immediately. However, I'm receiving reports of impossible flows:



I sorted the flows in descending order of TX and RX Bytes in those two images, respectively.

It's just not a possible amount of traffic. This isn't the first time I've encountered these crazy NetFlow statistics with Management Center, though. Last year, at a different organization, I was testing the feasibility of using Management Center to manage a Cisco network (it isn't very feasible, by the way), and I encountered similar impossible flows when I enabled reporting our our Cisco Catalyst 6513.

My evaluation license in Management Center expired before I discovered the source of those impossible flows.

The situation at my organization now is that we have fielded Summit X440 switches at remote locations, which are using FortiGate 100D firewalls as their gateways. From a flow monitoring perspective of network management, it makes sense to collect the flow data from the firewalls (which are capable of NetFlow v9, required by Management Center). The X440 switches are not able to report Netflow, and Management Center is unable to collect sFlow (which the X440 switches can report).

I've asked the Fortinet community about the possibility of filtering the NetFlow datagrams at the source, although their user community seems to be largely a ghost town.

Is it possible for me to ignore these flows at the Management Center? They are making half of the Analytics dashboard useless, by filling panes that sort by bandwidth with noise:



Any dashboard pane that reports TopN of anything by bandwidth shows those impossible flows, masking anything of use to me about our networks.

How to make Management Center report on flows only concerning our networks? Or, how to make it ignore that noise? Those are the problems I need to solve to make NetSight Analytics of any use to us.

21 replies

If NETFLOW is reporting it, I bet it's actually true and you have more of a problem than you think.
Also, my brain instantly screamed routing loop when I saw that.
Userlevel 1
You'd think, Jeremey. But, I'm not routing any of the source or destination addresses of those crazy flows. And, I have no latency or throughput issues with these remote sites. That detracts from the routing loop possibility.

What puzzles me is that I saw this same sort of crazy stuff from that other Cisco router I once configured to send NetFlow to Management Center.

This can't possibly be traffic through my monitored interface. The thing I can't get my mind around is from where is it coming?
Try doing some packet captures at the ingress of your network (where the firewall is). Could you make sure you have Null routes black holing everything else that you don't route for?
Userlevel 1
On page 853 of the Extreme Management Center User Guide, is a section called Flow Collector Filter:

Use this field to filter all incoming flows as they are processed by the flow
collector. Flows not matching the filter are discarded and not maintained in
memory on the server. If you add a filter here, the current flows stored in the
cache are trimmed to only matching flows.
Use this option if you want to use flow collection to look for specific results,
and unrelated flows do not need to be processed. For example, only
processing flows pertaining to a particular subnet."

That looks promising, but there's nothing at all in that manual describing the syntax to construct that filter.
Userlevel 1
I'll investigate those null routes, Jeremy.
This is what I use on our edge route.

ip route 10.0.0.0 255.0.0.0 Null0 200

ip route 169.254.0.0 255.255.0.0 Null0

ip route 192.168.0.0 255.255.0.0 Null0 200

ip route 172.16.0.0 255.240.0.0 Null0

You may need to adjust for your network.
Userlevel 1
Jeremy Gibbs wrote:

This is what I use on our edge route.

ip route 10.0.0.0 255.0.0.0 Null0 200

ip route 169.254.0.0 255.255.0.0 Null0

ip route 192.168.0.0 255.255.0.0 Null0 200

ip route 172.16.0.0 255.240.0.0 Null0

You may need to adjust for your network.

Jeremy, the clients and servers in those impossibly large flows aren't in any of my networks. What I mean to say here, I think, is that it isn't one of the non-routable networks as you've listed that are the sources of these flows I'm investigating.
Userlevel 6
Before we purchased Solarwinds we used this free sflow collector ... works great just does not create a data base for searches but for top talkers and such it is good... Vhttp://www.inmon.com/products/sFlowTrend.php .. This would give you the layer 2 traffic on the extreme side to compare to what you are getting on the firewall via netflow.
Userlevel 1
EtherMAN wrote:

Before we purchased Solarwinds we used this free sflow collector ... works great just does not create a data base for searches but for top talkers and such it is good... Vhttp://www.inmon.com/products/sFlowTrend.php .. This would give you the layer 2 traffic on the extreme side to compare to what you are getting on the firewall via netflow.

I'm collecting sFlow from the Extreme switches using PRTG as the collector. That output patches well what the NetFlow from the firewall reports, if I ignore the impossible flows I mentioned earlier.
Userlevel 1
On the Management Center > Analytics page, there exists a search/filter field, with which to operate on the Flows tab of that page. Contextual help shows a syntax for developing filters for that page:

Enter search criteria to filter table. One or more filters may be entered, separated by semi-colons. For each filter, individual components of that filter are comma separated. Specific flow components must be specified with a key. Keys are not case sensitive.

Supported keys: CLIENT, SERVER, SERVERPORT, PROTO, SENSOR, IF, INIF, OUTIF, APP, APPGROUP, META, LOCATION, DEVFAMILY, USER, PROFILE
*Note: CLIENT/SERVER may be an IP (CIDR Mask Supported) or host, and may contain a port (X.X.X.X:P)
Supported modifiers: duration, size, packets, bps, with a comparator of: >, >=, =, <, or <=
Supported keywords: Include/Exclude (Include is implied)
Fuzzy filters and partial hostnames are supported
*ie "Doe" will return flows w/ hostnames "JonDoe", "JillDoe", etc

Additionally flow metadata can be searched by using a colon as the key-value delimiter.
Example: meta=Content-Type:application/json

Everything is case insensitive.
A Fuzzy filter is a string without a keyword. It can partially match any value in any column. E.g. "Google" would match source or destination host name "www.google.com";, or application names "Google Mail" or "Google News".

Examples (single filter):
1) "client=JonDoe, server=10.20.77.33, snmp" –> SNMP Flows from host JonDoe to 10.20.77.33
2) "JonDoe, SENSOR=10.20.77.33, snmp" –> SNMP Flows from netflow sensor 10.20.77.33 To/From JonDoe
3) "SERVERPORT=161, duration>1000, exclude" –> All flows EXCEPT SNMP(161) flows that lasted more than 1 second
4) "snmp" –> All snmp traffic
5) "10.20.77.33:161" –> flows where server/serverport = 10.20.77.33:161
6) "JonDoe:161" –> snmp flows to or from JonDoe
7) app=DNS –> All flows identified as belonging to the application "DNS"
😎 app=89 –> All flows identified as belonging to the application whose ID is 89
9) appgroup=Social Networking –> All flows identified as belonging to the application group "Social Networking"
10) user=none,exclude –> All flows with an identified user
11) inif=11002 –> All flows where input interface = 11002
12) meta=snmp–> All flows with metadata containing the text SNMP
13) meta=Content-Type:application/json;> All flows with an application type of JSON

Example (multi filter):
1) "SERVERPORT=161, duration>1000, exclude; CLIENT=JonDoe; CLIENT=JillDoe" –> All flows from JonDoe and JillDoe except SNMP flows lasting more than 1 sec

If the filter fails to find a match, no data is displayed

Does anyone know if this filter syntax can be used in the Administration > Options > NetFlow Collection > Advanced > Flow Collector Filter field?
This also might be a good thread to look at...

https://supportforums.cisco.com/discussion/11141021/deny-hopopt-reverse-path-check-0000-0000-interface-inside
Better question, do you have reverse path check enabled on your FG?
Userlevel 1
Jeremy Gibbs wrote:

Better question, do you have reverse path check enabled on your FG?

Jeremy, I learned that Fortinet calls it reverse path forwarding, and it is enabled by default. There exists a command to permit asymmetric routing (set asymroute enable), which is not enabled on this firewall.

This mystery continues.
Userlevel 6
I met following behavior with Fortigate:
the Fortigate does cache netflow records. if the netflow destination is unreachable all records are cached. when the netflow destination is available again, you can get aggregated data.
can that explain your observation?
Userlevel 1
Pala, Zdenek wrote:

I met following behavior with Fortigate:
the Fortigate does cache netflow records. if the netflow destination is unreachable all records are cached. when the netflow destination is available again, you can get aggregated data.
can that explain your observation?

I wouldn't think so. Since I enabled Netflow on that fortigate, the collector (Extreme Management Center) has been available.

Even if not, those flows are describing volumes of traffic that wouldn't be possible for thousands of years over the size pipe to which the fortigate is connected. I really think it's a fault in how the Fortigate reported it, but I don't know how to either filter it out, or delete the information about those flows in Management Center.

My post on the Fortinet community remains unread and unanswered.

Here's a peculiar thing, though: The NetFlow sensor on our PRTG server never once reported these fantastic flows from our Fortigate firewalls.

With what I'm struggling is where lies the fault? Is the Fortigate reporting garbage data? If so, I can't figure why PRTG didn't display it, or, why Management Center analytics does display it.

I strongly suspect that Management Center is displaying what's been reported to it, and PRTG is filtering the NetFlow data before displaying it. Just a suspicion, though. I can't figure how PRTG would know what to filter out, and I didn't tell it to do so.
Userlevel 7
Pala, Zdenek wrote:

I met following behavior with Fortigate:
the Fortigate does cache netflow records. if the netflow destination is unreachable all records are cached. when the netflow destination is available again, you can get aggregated data.
can that explain your observation?

Hi Jesse,

did you try to take a packet capture of the Netflow data sent by the Fortigate, and looked at it using Wireshark?

The definitely impossible flows show neither input nor output interfaces. They do show some regularity. That might hint at some packet format not supported by Extreme Management Center and interpreted incorrectly.

HTH,
Erik
Userlevel 1
Pala, Zdenek wrote:

I met following behavior with Fortigate:
the Fortigate does cache netflow records. if the netflow destination is unreachable all records are cached. when the netflow destination is available again, you can get aggregated data.
can that explain your observation?

Hello, Erik.

I took your advice and captured 10000 NetFlow packets, which bracketed (in time) instances of impossible flows as shown again on the Analytics Dashboard.

None of the source IP addresses, destination IP addresses, or traffic protocols of the impossible flows displayed by the dashboard during the time period of my packet capture exist in the capture file.

I'm getting fairly good at using WireShark, looking for this stuff.

I think it's time to open a case with GTAC to investigate further.
Userlevel 7
Pala, Zdenek wrote:

I met following behavior with Fortigate:
the Fortigate does cache netflow records. if the netflow destination is unreachable all records are cached. when the netflow destination is available again, you can get aggregated data.
can that explain your observation?

Hello Jesse,

I suspect that some of the NetFlow packets use a format not supported by Extreme Management Center, but instead of dropping or ignoring them, EMC interprets them as the impossible flows you see.

Asking GTAC to take a look at this seems like a good idea. 🙂

Erik
Userlevel 1
Still having this problem. Can anyone point me to the syntax to use in the Administration > Options > NetFlow Collection > Advanced > Flow Collector Filter field?

I did a packet capture of all exports over a period of several hours, which bracketed some of the wildly impossible reports displayed in the Analytics Dashboard. No instances of any identifying piece of the offending dashboard graphs existed in that packet capture file.

This lends support to the suspicion that it is Extreme Management Center that is making the error.

However, if I can find out how to complete that filtering field, I can pare the NetFlow exports down to only those that describe traffic to or from my specific IP networks.
Userlevel 7
Hover over the search field with the mouse pointer till the info field show and click on "more" to get the options and examples....

Reply