InterVLAN Broadcast flooding problem

  • 0
  • 2
  • Problem
  • Updated 3 years ago
  • Solved
Hello, 
In our deployment we have a core switch (BD 8800) connecting to edge switches (x440-24p) through aggregation switches (x460-24x).
All the ports on edge switches are configured for at least two vlans, vlan 10 is voice and an untagged vlan for data or other applications.
Now the problem is I am seeing traffic (at least broadcast) from the untagged vlans appearing in voice vlan.
This is happening all over the network hence putting extra load on all ports and as a result the IP Phones are not able to acquire IP from DHCP server. If i remove the tagged vlan (i.e voice) from a specific port then the leakage from that port into voice vlan  stops.
Any idea about solving this issue ?
Photo of f3rha4n

f3rha4n

  • 230 Points 100 badge 2x thumb

Posted 3 years ago

  • 0
  • 2
Photo of Jarek

Jarek

  • 2,398 Points 2k badge 2x thumb
Hi,

did this work before ?
If you plug to the port only for example laptop and port config is vlan 10 tag + vlan untagged,
do you see this issue ?
Maybe the IP Phone is working as a bridge for tagged and untagged traffic and connect this two vlans...?

--
Jarek
Photo of Martin Flammia

Martin Flammia

  • 6,326 Points 5k badge 2x thumb

Wonder if somewhere you have a link with data untagged and voice tagged, yet the other end you have voice untagged and data tagged.

As both data and voice are connected untagged, I wonder if you can indirectly connect the vlans that way and therefore have broadcast spread across both?

Photo of f3rha4n

f3rha4n

  • 230 Points 100 badge 2x thumb
hi Jarek,
Currently my laptop is connected to an untagged voice vlan port on an x440 switch.
There is a device (10.154.0.22) connected on another switch which is currently looking for its server(10.154.0.10), the server is not installed yet.
The port to which the device is connected is tagged for voice vlan 10 and untagged for its data vlan 15. And i can see the ARP broadcast on my laptop, which as i mentioned earlier is connected to an untagged voice vlan port.
My goal right now is to eliminate any traffic in any vlan that does not belong to it, unless its routed by the core switch
 
Photo of f3rha4n

f3rha4n

  • 230 Points 100 badge 2x thumb
Hi Martin,
Nowhere in my deployment is voice untagged. All the IP phones detect the vlan through LLDP med (which i have disabled for now). The only untagged port for voice vlan is the one I am using for troubleshooting right now.
Photo of f3rha4n

f3rha4n

  • 230 Points 100 badge 2x thumb
here is my config for the edge switch where the traffic at port is getting mixed-up/leaked:
Test device is connected to port 19 which belongs to RMS vlan 15 for untagged traffic. All other vlans are tagged on that port.
172.16.4.22.8 # sh conf
#
# Module devmgr configuration.
#
configure snmp sysName "172.16.4.22"
configure snmp sysContact "support@extremenetworks.com, +1 888 257 3000"
configure sys-recovery-level switch reset

#
# Module vlan configuration.
#
configure vlan default delete ports all
configure vr VR-Default delete ports 1-24
configure vr VR-Default add ports 1-24
configure vlan default delete ports 1-24
create vlan "CCTV"
configure vlan CCTV tag 14
create vlan "Data-Admin"
configure vlan Data-Admin tag 12
create vlan "Data-Guest"
configure vlan Data-Guest tag 11
create vlan "IPTel"
configure vlan IPTel tag 10
create vlan "IPTV"
configure vlan IPTV tag 13
create vlan "net-mgmnt"
configure vlan net-mgmnt tag 20
create vlan "PA"
configure vlan PA tag 23
create vlan "RMS"
configure vlan RMS tag 15
create vlan "WAP-Mgmt"
configure vlan WAP-Mgmt tag 19
configure vlan CCTV add ports 1-24 tagged
configure vlan Data-Admin add ports 1-24 tagged
configure vlan Data-Guest add ports 20-24 tagged
configure vlan Data-Guest add ports 1-18 untagged
configure vlan IPTel add ports 1-24 tagged
configure vlan IPTV add ports 1-19, 24 tagged
configure vlan IPTV add ports 20-23 untagged
configure vlan net-mgmnt add ports 1-24 tagged
configure vlan PA add ports 1-24 tagged
configure vlan RMS add ports 1-18, 21-24 tagged
configure vlan RMS add ports 19 untagged
configure vlan WAP-Mgmt add ports 1-24 tagged
configure vlan Mgmt ipaddress 172.16.4.22 255.255.0.0
configure vlan net-mgmnt ipaddress 172.16.4.22 255.255.0.0

#
# Module fdb configuration.
#

#
# Module rtmgr configuration.
#
disable iproute ipv4 compression
disable iproute ipv6 compression

#
# Module mcmgr configuration.
#

#
# Module aaa configuration.
#
configure account admin encrypted "9FtorW$L4OVuc9.2rTtMC7x2AN4K1"

#
# Module acl configuration.
#



configure access-list zone SYSTEM application NetLogin application-priority 4
configure access-list zone SYSTEM application HealthCheckLAG application-priority 5
configure access-list zone SYSTEM application IdentityManager application-priority 6
configure access-list zone SYSTEM application VMTracking application-priority 7
configure access-list zone SYSTEM application PolicyManager application-priority 8
configure access-list zone SYSTEM application Snmp application-priority 11
configure access-list zone SYSTEM application Telnet application-priority 12
configure access-list zone SYSTEM application Http application-priority 13
configure access-list zone SYSTEM application Ssh2 application-priority 14

#
# Module bfd configuration.
#

#
# Module ces configuration.
#

#
# Module cfgmgr configuration.
#

#
# Module dosprotect configuration.
#

#
# Module dot1ag configuration.
#

#
# Module eaps configuration.
#

#
# Module edp configuration.
#

#
# Module elrp configuration.
#

#
# Module ems configuration.
#

#
# Module epm configuration.
#

#
# Module erps configuration.
#

#
# Module esrp configuration.
#

#
# Module ethoam configuration.
#

#
# Module etmon configuration.
#

#
# Module hal configuration.
#

#
# Module idMgr configuration.
#

#
# Module ipSecurity configuration.
#

#
# Module ipfix configuration.
#

#
# Module lldp configuration.
#

#
# Module mrp configuration.
#

#
# Module msdp configuration.
#

#
# Module netLogin configuration.
#

#
# Module netTools configuration.
#

#
# Module ntp configuration.
#

#
# Module poe configuration.
#

#
# Module rip configuration.
#

#
# Module ripng configuration.
#

#
# Module snmpMaster configuration.
#

#
# Module stp configuration.
#

#
# Module synce configuration.
#

#
# Module techSupport configuration.
#

#
# Module telnetd configuration.
#

#
# Module tftpd configuration.
#

#
# Module thttpd configuration.
#
configure ssl certificate hash-algorithm sha512

#
# Module vmt configuration.
#

#
# Module vsm configuration.
#
172.16.4.22.9 #
Photo of Martin Flammia

Martin Flammia

  • 6,326 Points 5k badge 2x thumb

Just to be clear the device 10.154.0.22 I assume is in your Data Vlan right? and your seeing ARP's from this device even though you connected directly into the Voice Vlan?

I cant imagine any other way how layer 2 broadcasts could hop vlans unless they are not connected in someway. Obviously the method I mentioned could be happening / configured anywhere in your network and you would still see it where you are connected.

If not that then I'm stumped at the moment.

Photo of f3rha4n

f3rha4n

  • 230 Points 100 badge 2x thumb
yes you are right 10.154.x.x is data vlan,  and I have shutdown all the other switches. only 4 switches are ON in total. The scenario right now is like:
device-->[edge sw]-->[aggregation sw]-->[core sw]<--[edge sw]<---my laptop 
the only untagged ports right now are the 2 ports, 1st whre the test device is connected and 2nd where my laptop is connected. Both of these are on different vlans. 
The configuration across all the switches is similar so I dont know where the mixup can possibly happen.
And as i mentioned earlier if i remove for example voice vlan from that test port then i stop seeing the broadcast from that device in voice vlan. So as far as i can see the mixup is happening on source port.
(Edited)
Photo of f3rha4n

f3rha4n

  • 230 Points 100 badge 2x thumb
UPDATE: I disconnected the switch where my laptop is connected from rest of the network.Then i connected another laptop to another port which is configured for untagged data and tagged voice on this same switch.Then I started sending ping request to an unavailable address and I can still see the ARP requested generated by that laptop in data vlan on my laptop which is in voice vlan.
scenario right now:
test laptop--->[switch]<---my laptop
Photo of Jarek

Jarek

  • 2,398 Points 2k badge 2x thumb
Do you use feature like "VLAN Translation" ?
Photo of Martin Flammia

Martin Flammia

  • 6,326 Points 5k badge 2x thumb

Whether this has any bearing or not - What's the full mac address in your capture, is it a multicast address? Not that it should still traverse a Vlan with your configuration.

Wonder if your not actually seeing traffic transverse Vlans but something else related to the NLB?

Photo of f3rha4n

f3rha4n

  • 230 Points 100 badge 2x thumb
the source address always appears of the device generating the broadcast traffic and destination mac is always 00:00:00:00:00:00. So far I have only reproduced this issue using ARP.
Photo of Paul Russo

Paul Russo, Alum

  • 9,694 Points 5k badge 2x thumb
Hello Ferhan  In your original test you had a device that was on both the data and voice VLAN and your PC was on the data VLAN.  The test device was looking for a server that was not there but the IP address on the capture were on the same VLAN.  Was the device actually using the TAG for the voice VLAN?

I have seen something like this before where the device that was supposed to be on the voice VLAN came up and originally since it can't get to its server will come up on the untag VLAN.  Most voice applications the phone first comes up with a DHCP address on the data and then is told, using LLDP or DHCP, to move over to the voice VLAN and use the tag.

Try this test make the port that the phone is on strictly Voice and tag it to work on the voice VLAN then see if you see the broadcast on data.  that would show if broadcast are "leaking" between VLANs or if you are just seeing the voice device sending broadcast on the data VLAN.

Thanks
P
Photo of f3rha4n

f3rha4n

  • 230 Points 100 badge 2x thumb
Hi Paul,The device can only use untagged traffic and thats what it was doing. The IP phones are from Mitel and they detect voice vlan using LLDP MED, which they were successfully doing. The problem arose when they tried to acquire IP from DHCP server and the DHCP DISCOVER messages  somehow got lost in all that broadcast traffic and the phones were not able to reach the server. 
I will try to elaborate the problem. The issue is not specifically related to voice vlan only.

Test Device is connected to a port 1 which is configured to be tagged for vlan A B C D and untagged for vlan X,
I connect my laptop to port 2 which is configured to be tagged for vlan B C D X and untagged for vlan A.
In this scenario my laptop (at port 2) shouldnt be able to see traffic on vlan A originating from untagged vlan X on port 1 but when i run wireshark I can clearly see the broadcast. 
Same happens if i use any other vlan.
Photo of EtherMAN

EtherMAN, Embassador

  • 7,370 Points 5k badge 2x thumb
Correct me if I am understanding this wrong but no matter if you have tagged or untagged vlans on the port you plug the laptop into the laptop will be presented with any and all Mcast frames, broadcast frames, unknown mac address frames for all vlans on that port.  So you will be seeing packets from other vlans.  
Photo of Martin Flammia

Martin Flammia

  • 6,326 Points 5k badge 2x thumb

Hi Ferhan - appreciate what you are saying, know Mitel very well, but just wondering if your PC therefore gets an IP address when its connected to the Voice Vlan only, untagged.

I know that doesn't address the problem your are outlining but interested in the result based the description you have given regarding the initial phone problem.

Thanks.

Photo of Paul Russo

Paul Russo, Alum

  • 9,694 Points 5k badge 2x thumb
EtherMAN you are correct.  This is the same as using IP multinetting.  If a port is a part of 4 VLANs all the broadcast for those VLANs will go out the port.  The switch keeps the broadcasts within the VLAN and sends it to all ports that are part of that VLAN regardless of how many other VLANs are on there.

If you are doing wireshark and it is in promiscuous mode then you will see everything.

If the issue however is that port 1 is on VLAN A and port 2 is on VLAN B and you are seeing VLAN As broadcast on port 2 then that is an issue as both VLANs are not on both ports.

The question I think is why are there so many broadcast that it is affecting your devices from getting DHCP.  You can use the port broadcast threshold option to restrict how much broadcast gets sent across the VLAN.

P
Photo of f3rha4n

f3rha4n

  • 230 Points 100 badge 2x thumb
@EtherMAN: you are correct
@Martin: when i connected PC to an untagged voice vlan port it got the IP easily and I could see the dhcp discover packet at Mitel PABX, But when I plugged Mitel Phone into that untagged port (and turned LLDP off) then it just kept on sending dhcp discover message that I could see on wireshark but they most probably never made it to the PABX, which bugged me. This was when a broadcast generating single device was on the network.
Also to me it appeared that the edge switches and Core switch had problem with populating CAM table or something like that because initially when I pinged the PABX (or other devices) then the arp seemingly didnt reach the destination and I never got reply, then i pinged the core switch from PABX and after that the PABX became pingable from core switch and after a minutes or so from other locations. this happened many times but that is not my immediate concern for now.

@Paul: I dont remember running wireshark in promiscuous mode but now that i checked its running in promiscuous mode by default. Anyway in that case shouldnt I be able to see the vlan tag in packets? because i tried to look for it and couldnt find any.

For now I resolved the DHCP issue by removing all other vlans from PABX port and putting it on untagged voice only. But I am still interested in reducing broadcast so please tell me how can I accomplish that ? 
Also I wondered myself about the amount of broadcast being generated and its effect on dhcp, I only enabled one device throughout the network and shutdown all other switches and that one device successfully blocked IP Phones' attempt to reach DHCP server on PABX, you can see a screenshot that I posted earlier.

Thanks
Photo of Martin Flammia

Martin Flammia

  • 6,326 Points 5k badge 2x thumb
Actually it's funny you should say that as I've seen this before when the FDB table doesn't insert the MAC address of the device until you make it send some traffic, like ping, be interested to see if anyone comes up with an answer on that.

Couple of things that would be interesting to try, the first would be to statically configure the phone with an IP, gateway, controller IP, vlan I'd etc and see if it starts working.

The other would be to add a helper address / bootrelay command on the voice vlan pointing to your PBX that's acting as the DHCP server for your voice vlan. I know you shouldn't need it on the same vlan but could be worth a try.

Not personally tried Mitel phones on the purple kit before, mainly the red.
Photo of Martin Flammia

Martin Flammia

  • 6,326 Points 5k badge 2x thumb
One other thing regarding seeing the 802.1p/q tag in wireshark, this is very common as a lot of nic's don't support tagging. Some network cards you can enable it in the nic settings or you have to change something in the registry. On my laptop for example my internal mic doesn't support it so I use an external USB Ethernet module that does.
Photo of f3rha4n

f3rha4n

  • 230 Points 100 badge 2x thumb
I did try bootprelay before starting this thread and that didnt help.But as I said earlier i have now resolved the dhcp issue. Right now I am only interested in reducing the broadcast being generated at the source ports because apparently that is too much for extreme switches and mitel phones to handle properly (even if the source of broadcast is a single device in entire network).
Photo of Martin Flammia

Martin Flammia

  • 6,326 Points 5k badge 2x thumb
The only other thing that I can think of that would create that much broadcast traffic of the same type from a single device across two different vlans is that you have a loop somewhere. I know that doesn't seem possible in the scenario you have given but might be worth turning on spanning tree / elrp ?
Photo of f3rha4n

f3rha4n

  • 230 Points 100 badge 2x thumb
Martin I think you didnt notice but we already found the root cause of traffic appearing to be in different vlan. It was not actually in different vlan but because wireshark was in promiscuous mode that is why it was seeing those packets from the source vlan too (which was tagged on the port where i was monitoring it). Although I am still confused about why I was not able to see any tags in packets if wireshark was reading all the raw data.And I reduced the test scenario to a single single switch and disconnected it from the network so any chance of looping etc is out of the question.
Thanks for your suggestions but right now I am focused on reducing the broadcast at source port.
Photo of bw447

bw447

  • 966 Points 500 badge 2x thumb
@Ferhan: Are you still seeing high broadcasts on your source port?