Frequent ARP request broadcast for Who has x.x.255.255? from S-Series root switch in RSTP VLAN

  • 0
  • 1
  • Question
  • Updated 2 years ago
  • Answered

We are getting average of 1 pps and peaks of 60 pps of these packets, from Wireshark trace.

Please kindly share information on what is the cause of these broadcasts and how to minimize the frequency.

TIA.

Screenshots of Wireshark decode and IO graph below:





Photo of Edwin

Edwin

  • 156 Points 100 badge 2x thumb

Posted 2 years ago

  • 0
  • 1
Photo of Mel78, CISSP, ECE

Mel78, CISSP, ECE

  • 1,044 Points 1k badge 2x thumb

Theres nothing i can do when you mask out the source ip of the broadcast storm.

Likely is a STP loop that causes broadcast storm.

You can use ELRP to protect your own switch but you can never eliminate the root cause if you cannot block the source. 

Photo of Edwin

Edwin

  • 156 Points 100 badge 2x thumb
Sorry, the source IP addresses are for the root and backup root S-series switches for the RSTP VLAN network.
Photo of Mel78, CISSP, ECE

Mel78, CISSP, ECE

  • 1,044 Points 1k badge 2x thumb
Then likely is a loop. Do you have 2 or more parallel patch cords connected betwwen the root switch and backup root switch ?
Photo of Edwin

Edwin

  • 156 Points 100 badge 2x thumb
I will check on the parallel patch cords next week WPL; it's weekend her now.

Just a side query please, what is the probable cause or reason for an ARP request from the root and backup root switches for an IP address that is actually the subnet mask for our RSTP VLAN, and on a regular or periodic basis? Will all hosts in the subnet respond to these broadcasts? The Wireshark trace captured only broadcasts and unicasts to and from the PC running Wireshark.

Thanks.
Photo of Edwin

Edwin

  • 156 Points 100 badge 2x thumb
Just found an article from Juniper.net,

http://www.juniper.net/documentation/...

which details the mechanics of IP Directed Broadcast from a remote administration task (such as a backup server) to the hosts in a specified subnet.

We have several servers north of the RSTP VLAN in a firewalled DMZ, that handle backup and patch management tasks for the workstations in the subject subnet. So, perhaps the root and backup root switches are forwarding the remote IP Directed Broadcasts as ARP requests for Who has x.x.255.255 in the subject subnet? How best to check / confirm this? Mirror the suspect interfaces to the Wireshark PC interface, perhaps and collect and analyze traces?
Photo of Edwin

Edwin

  • 156 Points 100 badge 2x thumb

The root and backup root switches have 2 interconnecting patch cords.

The root and backup-root switches were recently upgraded from N-series to S-series, and we did not have these ARP broadcasts for x.x.255.255 earlier.

Checking the Wireshark trace, I find that the ARP requests for x.x.255.255 are being generated right after the NTP broadcasts to the same x.x.255.255 address. The SNTP Server has a Satellite GPS timesync card and is on one of the subnets.

Does the label, "all-subnets-directed broadcast" apply to the NTP broadcasts, even though the source is on one of the subnets?

The Wireshark IO graph (using stacked bars) is shown below; with the NTPs in blue at 2 pkts every 16 seconds, ARPs from root switch in yellow, and ARPs from backup root switch in purple:

Why are there 6 sets of ARPs per switch for every other NTP?

Your advice and guidance on this matter is very much appreciated.


Photo of Mel78, CISSP, ECE

Mel78, CISSP, ECE

  • 1,044 Points 1k badge 2x thumb
Hi,

I do not want to speculate any further. You have identify the source of the ARP broadcast which is the SNTP server time sync via broadcast.

So either you have to segregate different vlans to limit the broadcast domain and use the switch as proxy arp or there are countless ways to mitigate the issues. But you cannot eliminate the root cause if you allow NTP server to broadcast.

Just another point to add, the source of the broadcast from NTP server is likely because the NTP server did not configure any gateway. Once theres any NTP request (unicast) to the NTP server, the server will "SHOUT" out to everyone. And your switches will echo everywhere and thus broadcast storm but low traffic.
(Edited)
Photo of Erik Auerswald

Erik Auerswald, Embassador

  • 12,782 Points 10k badge 2x thumb
Hi,

IMHO, the switches should not create ARP requests for the subnet broadcast address.

You might want to enable directed broadcasts on the S-Series, since your SNTP server uses them:
configure
interface vlan.0.XXX
ip directed-broadcast
The output of "show ip interface" tells you if directed broadcasts are enabled on an SVI.

Erik
Photo of Edwin

Edwin

  • 156 Points 100 badge 2x thumb

Thank you Erik,

Do we know why the ARP requests for the subnet broadcast address are particular to the S-series?

We started seeing these packets only after replacing the N-series root and backup root switches under an upgrade program.

The N-series Configuration Guide states that IP-directed broadcasts are disabled in the default setting -- the configuration files did not have any reference to directed-broadcast.

The start of the N-series root switch config file is as shown below; I am still trying to secure copy of the config file for the S-series.

===================================================================

## Config File was imported 12-14-2015 09:24:21 from SWROOT (7C103) N3 Chassis ##

set banner motd "Configuration imported from SWROOT on 12-14-2015 09:24:21 running firmware rev DFE-P-6123-0003"
set ip address x.x.81.1 mask 255.255.0.0
set ip route default x.X.81.1

====================================================================


Photo of Erik Auerswald

Erik Auerswald, Embassador

  • 12,782 Points 10k badge 2x thumb
Hi Edwin,

that N-Series configuration pertains to the switch ip address, not a router (SVI) address. That interface does not forward IP packets. The default route pointing to the switch management interface is surprising.

Firmware version 6 was the last version to use different IP stacks for switch management and routing / layer 3 forwarding. The S-Series never had firmware version 6, it started with version 7 using a unified IP stack. This new IP stack might cause the different behaviour.

Did you have a router configuration on the N-Series switches? Do you want the S-Series to forward IP packets?

I understand the situation as follows:
  1. Both S-Series receive the directed broadcast packet sent with a broadcast MAC address.
  2. Both S-Series want to forward the packet to its destination.
  3. Both S-Series have the IP subnet as directly connected in there routing tables, so they try to deliver the packet locally.
  4. For local delivery, the S-Series need to know the MAC address for the x.x.255.255 IP address. This address is not found in the ARP table, so ARP requests are generated. Those are repeated, because there is no reply. (Those ARP requests should not be generated.)
  5. Neither S-Series forwards the directed broadcast packet.
I suppose your network is OK and you just want to get rid of the ARP requests?

You might want to consider disabling ip forwarding on the S-Series' management interfaces, if you do not want it to act as a router and forward IP packets.

Erik
Photo of Edwin

Edwin

  • 156 Points 100 badge 2x thumb

Hi Erik,

Our RSTP with VLAN-2 Class B network with subnet mask of 255.255.0.0 is normally used offline from the Internet and data or comms links are confined only to the subnets with the all known dual-homed hosts belonging to VLAN-2. Normally, no routing requirements. This was the environment with the N-series root and backup root switches.

With the upgrade, the S-Series were adopted, together with the addition of cyber security measures, such as, PDC/SDC, WSUS, ePO and backup servers. These servers sit inside a DMZ above the RSTP network. These servers connect to the (additional) 3rd NIC of the RSTP hosts. Traffic is isolated from the original 2 (fault tolerant) NICs. I believe the "3rd Ethernet" network has also its own VLAN and subnet assignments. 

The SNTP server (with satellite GPS card) remains inside the RSTP network.

The new network is operating ok, and we just want to check if we can get rid of the ARP requests (for x.x.255.255), which as you mentioned, are not getting any response.

Is there anyway to validate the assumption that the S-Series switches are generating the subject ARPs due to the NTP time sync broadcasts to x.x.255.255; besides switching off the SNTP host machine? Why are the ARPs generated after every other set of 2 NTP broadcasts -- there are 2 NTP pkts every 16 seconds, the ARPs only occur for one set (repeating only every 32 seconds). Can there be other sources besides the NTP broadcasts, which our Wireshark traces are not able to capture (without port mirroring, etc.)?


Photo of Edwin

Edwin

  • 156 Points 100 badge 2x thumb
PS. All known hosts have static pre-assigned IP addresses and host names are defined inside the "hosts" files.
Photo of Erik Auerswald

Erik Auerswald, Embassador

  • 12,782 Points 10k badge 2x thumb
As a quick test you could block SNTP on an S-Series using an ACL. You would lose time synchronization for that switch for the duration of this test.

Another possible test would be to generate a different directed broadcast packet and see if ARP requests are generated (note that ARP request generation is rate limited).

If there are other packets sent to the local broadcast address x.x.255.255, those should result in ARP requests from the S-Series as well.

You might want to take a look at the following GTAC Knowledge article (the 7100 switches use the same EOS as the S and K Series):

https://gtacknowledge.extremenetworks.com/articles/Solution/7100-series-switch-sending-syslog-messag...

I think you do not want the S-Series switches to act as routers. If the S-Series have interfaces in both the VLAN-2 and the DMZ, by default they would forward IP packets between both networks.

BTW, enabling directed broadcasts on SVIs is needed to send a directed broadcast from a different network to the SVI's subnet only. As I understand it the directed broadcast is generated inside the network it is destined for, no forwarding needed.
Photo of Edwin

Edwin

  • 156 Points 100 badge 2x thumb

As additional fyi that might provide more clarity for this query; the following are extracts from S-Series root switch config:

# ***** NON-DEFAULT CONFIGURATION *****
!
# Chassis Firmware Revision:  08.32.02.0008
!
#  SLOT   TYPE
#  ___    ________________
!
#   1     SSA-G8018-0652
!
# modal configuration
!
configure terminal
!
 interface vlan.0.2
  ip address x.x.81.1 255.255.0.0 primary
  no ip proxy-arp
  no ip forwarding
  no shutdown
  exit
!
# ip interface
set ip interface vlan.0.2 default

Looks like no IP forwarding and the S-series switch does not have interface in the DMZ, only in VLAN-2.

There is also "no ip directed-broadcast" setting found in the "show config all"; so this feature is disabled, by default.

Can I use "ping x.x.255.255" to see if similar ARP requests are generated?

Looks like we just have to live with these subject ARPs.


(Edited)