IPV6 broadcast storm, on an IPV4 network

  • 0
  • 1
  • Question
  • Updated 1 year ago
  • Answered
  • (Edited)
We have a number of printers that by default have IPV6 configured though not used. ( not supported by me )
These printers are also   connected to an IPV4 vlan where the IP address is allocated via DHCP and routed at a distribution switch X460.

Recently we had a issue with only the printers on the network , other devices on the same network , were able to operate and use the dhcp service and access the internet.

Only these printers would not work , if we connected a laptop to the switch port or disconnected the printer and used that port , the laptop would operate as expected.

We are being informed by the printer manufacturer that the issue we are seeing is that the Printer is creating an IPV6 broadcast storm , which I could understand , though my question is :

Would this not present itself the same as an IPV4 broadcast storm , CPU BCMrx high , vlan unresponsive etc?

This might be a IPV6 multicast issue as well ..

I.E the IPV4 network would also be effected .. which it was not.

I look forward to any comments and or suggestions .

Regards
Photo of Rod Robertson

Rod Robertson

  • 2,354 Points 2k badge 2x thumb

Posted 3 years ago

  • 0
  • 1
Photo of Patrick Voss

Patrick Voss, Alum

  • 11,714 Points 10k badge 2x thumb
Hi Rod,

How high is BCMRX getting in the "top" output? Is the issue still happening or is there a workaround to resolve it? If the issue is resolvable how long before it comes back?

There is a way to look at the packets hitting the CPU but this requires a case with GTAC and debug-mode to be enabled. Pending valid entitlement on the switch this may be the best route moving forward.
Photo of Mike D

Mike D, Alum

  • 3,852 Points 3k badge 2x thumb

Hello Rod,

fwiw, I agree with your doubt about the diagnosis.  Also agree with your reasoning - a traffic flood problem would leave tracks you would be able to follow.

On the flip side - no help understanding what's really going on.  

Regards,

Mike

Photo of Patrick Voss

Patrick Voss, Alum

  • 11,714 Points 10k badge 2x thumb
Hi Rod,

I took a second look at you post and misunderstood your question. High CPU will only appear if there is a lot learning going on. This traffic is automatically sent to the CPU for processing. If the traffic is being switched normally then you wouldn't see an issue unless the port is being over utilized. Without knowing what kind of traffic this is it would be very difficult to understand why this is only affecting the printers. Maybe you can mirror one of the printer ports and do a packet capture.
Photo of EtherMAN

EtherMAN, Embassador

  • 7,370 Points 5k badge 2x thumb
wonder if this is related to mcast listening floods we had back in 2014 when intel had bad drivers and if IPV6 were enabled and the nic card went to sleep.... ?? 
Photo of Derek Brown

Derek Brown

  • 170 Points 100 badge 2x thumb
The issue is not a BROADCAST store. IPv6 does not use Broadcast, only multicast. Since we are not using IPv6 on the network yet I can't be of any more help than to eliminate what the vendor said so you aren't going down a wasted path. The first thing I ALWAY do when there is a problem like this is mirror the port and capture the data with Sniffer or Wireshark. Only then can you see what the traffic is and what it is really doing.
Photo of Rod Robertson

Rod Robertson

  • 2,354 Points 2k badge 2x thumb
Hi many thanks for your comments , the engineers on site did take a trace and found that this is as suggested an IPV6 Multicast Listener Report ( STORM) .. so it would seem that the following scenario exists.

As we can have multiple networks on a single "piece of wire " ( old school ) we have the standard IPV4 behaving as expected and we also  have the IPV6 ( Printer Default configuration ) sharing the same physical wire , though on a separate network , because all the printers are on the IPV6 network as well as the IPV4  , when one of them starts this IPV6 Query the others response and this seems to start a cyclic process that is "self destruction".

It has  been suggested on other forums that Storm control , and other edge configurations could help , the main one being disabling IPV6 on the printers.

So I seem to have a fix ,Though  I still do not understand how the IPV6 packets , hitting the Switch CPU , would not register as high BCMrx and then be dropped or were theses IPV6 packets hitting another process , and therefore we missed this .

Any advise on how IPV6 hits the cpu and the process used would be appreciated ..

Many Thanks again
Photo of Patrick Voss

Patrick Voss, Alum

  • 11,714 Points 10k badge 2x thumb
Hi Rod,

How are we confirming that they are indeed hitting the CPU? It seems very odd to me that only the printers that are producing the traffic are experiencing issues even though there are multiple VLANs traversing the same physical interfaces. I do not think there is enough information to provide an explanation for what is happening on this network.
Photo of Jarek

Jarek

  • 2,398 Points 2k badge 2x thumb
Hi,

maybe you have arp storm or other traffic that is hitting the cpu? Wireshark is your friend as Derek suggested.

You can also block ipv6 traffiic via vlan acl if you want.
--
Jarek
Photo of Stephen Elliott

Stephen Elliott

  • 1,244 Points 1k badge 2x thumb

we've had exactly the same issue. 

as mentioned above it's not a broadcast storm (often caused when there's a loop in the network) but a lot of spurious IPV6 multicast packets being sent from one or two machines with a particular NIC (intel i217's from memory). 

I think the switches and other non-v6 devices can cope perfectly well with the amount of traffic being forwarded, but the printers themselves get overloaded and stop functioning correctly, perhaps because they are the only devices listening to IPv6 multicasts?

if an actual broadcast/multicast storm was happening then I believe the switches would also start to suffer.

Photo of Rod Robertson

Rod Robertson

  • 2,354 Points 2k badge 2x thumb
Hi

Thanks for the updates ..

We have a wireshark trace and have identified a number of "IPV6 top senders as previously suggested.

We are also going to control the IPv6 multicast by ACL.

As suggested in the posts , I think the Printer issues are only an " indicator" that there is an IPV6 multicast issue on the  network, and we have seen no immediate effect on network performance.

Again thansk for all your comments and suggestions.,
Photo of Mike D

Mike D, Alum

  • 3,852 Points 3k badge 2x thumb
Hello Rod,

Yours wont be the last time the community runs across this or related behavior. 

Pain for your network will become a signpost in another troubleshoot.  This forum is searchable from google - I'll also add a knowledge base doc.

First time I've personally heard about this sort of issue and isolated printer device-type behavior.    On the other hand I learn networking on this forum daily.

Thank you for taking a minute to follow up. 

Best regards,

Mike
Photo of Haris khan

Haris khan

  • 60 Points
Hello All,

Did you guys find solution for above problem as i am facing same issue on my network now. I captured packets from Wire shark software and found 5 desktop systems in my network who are broadcasting IPv6 packets in network. Right now i am disabling IPv6 TCP/IP option in those desktop systems lets see what will happen!!!
Photo of Erik Auerswald

Erik Auerswald, Embassador

  • 13,792 Points 10k badge 2x thumb
You could write an ACL (EXOS) or policy (EOS&EXOS) to drop any IPv6 packets at the edge ports by filtering on the IPv6 Ethertype of 0x86DD.

But this is a problem with the client devices (desktop systems in the recent case, printers in the older one) that should be addresses there.

BTW, since IPv6 uses multicast, non-IPv6 devices are usually less affected by misbehaving end systems than was (still is) the case with IPv4.
Photo of Stephen Elliott

Stephen Elliott

  • 1,244 Points 1k badge 2x thumb
Hi, the fix for the root cause is to update the Intel nic drivers on the affected machines.