How to Diagnose and Troubleshoot Unicast Flooding

[FAQ_User]

Article ID: 5322

Goals
How to troubleshoot unicast flooding
Find mac address

Symptoms
Unicast flooding
Seeing half of conversation in sniffer trace

Cause
From the perspective of a passive sniffer, a non-mirrored switch port should be seen to transmit just broadcasts, multicasts, and unknowns. If sorting a sniffer Host Table analysis (in, for example, Sniffer Pro) by "In Packets", shows that any station that received packets was not seen to transmit them, or vice versa; you likely have a unicast flooding problem.

Though unicast flooding will not prevent delivery of network traffic, it can be a concern from both efficiency and security perspectives.

Unicast flooding on switches running in traditional mode is normally due to the lack of a Source Address Table (SAT) entry on the switch, for the destination MAC. With an unknown destination, the packet is flooded. The reasons for the lack of SATable entry, on the other hand, are not quite as clear-cut.

Possible flooding scenarios:

Loadsharing NICs or servers
using asymmetrical transmit and receive policies. It is not a simple matter to predict Loadsharing NIC characteristics/operation, as there are a variety of similar products in existence which are designed for load sharing and redundancy. Each of them, in turn, may have a variety of configuration options which change their functional characteristics. Many of these products do not operate well (the attached switch fails to learn its forwarding information, or mis-learns it, or continually re-learns it on different trunk ports) in a non-proprietary switched environment, but some do. Given the variety of possibilities and the fact that these products, as with just about anything in the networking world, are always evolving and thus present a moving target; we have made no comprehensive attempt to document them. However, if functional details can be provided, such as whether each of the NICs in each cluster transmits and/or receives, the source MAC structure on layer 2 of each transmitted packed, and the source MAC/IP structure on layers 3 & 4 of each transmitted packet, we can assess how well they would function with switches not specifically designed to accommodate them. In the absence of these details, then a perhaps 30-second Sniffer Pro-readable sniffer trace local to each of the separate NICs in the cluster should tell the tale. UDP traffic
doesn't require acknowledgements, so knowledge of the destination MAC could time out during a long file transfer.
Multiple routers in a redundantly routed network
can send both ends of a conversation through different paths, resulting in an asymmetric flow of traffic, isolating some switches from knowledge of certain end stations.
• Using both a centrally based router serving as a default gateway,and a firewall router that also has direct access to the local stations' network; what can happen is that outgoing traffic will first hit the central router, which will map to the firewall router, and out of the network. The return trip will first go to the firewall router, which will then talk directly to the local station, bypassing the central router.
• When using a standby router protocol such as VRRP or HSRP, even if there is no failover, there are not necessarily any guarantees (especially in a multipath, equal-cost scenario) as to how the traffic will be returned from the remote side of load-sharing routers. If the "wrong" path is chosen, we again have an asymmetric flow.
Additional points to consider, regarding layer 3 behavior:
• If in examining the layer 2 information of flooded frames, either source or destination MAC consistently references a router; or, in examining layer 3 you find that the source and destination networks consistently differ; or you see multiple router MACs bound to a single IP address; then asymmetrical routing is a good possibility.
• Is there more than one layer 3 router with direct access to the layer 2 fabric in question? A simple test would be to examine primary and secondary IP network numbers for all router ports in the network.
• IP Routers generally, by default, send out an ICMP Redirect frame to an originating station, if they know of a more preferable path than themselves. Do the routers have ICMP redirect enabled, and are they running a routing awareness protocol such as RIP? What are the statically configured routes? If an end station has a self-referencing default gateway, it will ignore ICMP redirects. In that event, it will also need a router running proxy ARP, in order to communicate out of the local network.
Asymmetrical VLAN flow / Port Overlapping
(cross-vlan, within a switch) using separate FIDs for each VLAN. Unicast flooding would be likely if the VLAN used in one direction of traffic flow differs from the VLAN used in the opposite direction of traffic flow.
Overflowing Source Address tables.
Most devices have a capacity for at least 4000 addresses.
STP Aging Time (

dot1dTpAgingTime

) is consistently being flushed
down from 300 seconds to 15 seconds, as a result of Spanning Tree Topology Change events.
Solution
Possibilities:
• Use Loadsharing/Teaming software that utilizes a Multicast (rather than Unicast) destination MAC, and scope that traffic to a VLAN or to a set of ports (5708).
• Ensure that the firewall router goes back through the central router in order to access the local network.
• Plug both routers into the same switch.
• Take advantage of the fact that both routers and end stations have to keep an active ARP entry of the next hop. If you set the ARP timeout of the router (20 minutes by default, on the X-Pedition router) or affected end station (10 minutes by default, on UNIX machines) similar to the SAT timeout of the switches (usually 5 minutes by default), there will be sufficient knowledge of the whereabouts of end stations to prevent flooding.
• Set up a UNIX

  cron

  job to periodically ping out of the "silent" station (using the "silent" MAC!) to a non-existent IP host on its subnet, thereby ARPing to all devices, making them aware of its location.
• Set up a UNIX

  cron

  job to periodically ping the "silent" station.
• Use a common FID - SVL mode - among VLANs that share traffic within a switch.
Supporting documentation:
• 2839, "Flooding Unicast Traffic due to Load-sharing NICs"
• 4918, "In 802.1Q, what is the difference between IVL and SVL?
• 3460, "Changing Source Address Time-out on Switches and Bridges"
• For a switch, use SPEL/EMAN's Device / Find Source Address to query for a MAC address. The SATable source port will blink.
• For a switch, use SPEL/EMAN's Device / Bridge Status / Bridge / Filtering Databaseto view the SATable, and/or to configure a 'Static Filtering Database' entry.
• For a SmartSwitch 2nd/3rd Gen running 5.01.33 firmware or higher, the

  Network Tools

  command '

  show mac

  ' can be used to view the SAT, including FID identifier; plus it has sub-commands for granularity.
• Use SPEL/EMAN's Device / System Group / Other Groups / Net to Media Table to view the ARP Cache, aka ipNetToMediaTable (this might help).
• For a SmartSwitch, the

  Network Tools

  command '

  show ip

  ' can be used to view the ARP Cache, aka ipNetToMediaTable, plus other misc data.