mDNS Flooding when Inter-Station Traffic and Bonjour enabled for Apple Classroom integration.

[it70]

Hey everyone! I am working with support on this issue, but wanted to get some community thoughts. I an the network administrator at a private school and we have a 1-to-1 iPad solution for students and faculty. They utilize Apple Classroom for management. I have Inter-Station traffic enabled and Bonjour Gateway configured to allow discovery traffic. Problem is, this has been creating huge amounts of mDNS flooding, so much that it is overwhelming some APs to the point of losing HiveManager connection. The obvious thing to try would be to disable the "Inter-Station Traffic" setting (which I am going to test). But was wondering your guys thoughts on this. Has anyone else used Aerohive with Apple Classroom or might have a work-around?

[it70 · Accepted solution]

Alright, after this much time, the issue has been resolved. Below are the steps I have taken to mitigate the issue. Before all of our testing, we were seeing mDNS CPU usage up above 65%, now it is not even reaching 3%. The number of packets captured within a span of 5 minutes was originally around 800,000 with 95% of which were just mDNS queries and responses. Now, that number in the same amount of time is around 60,000 with only 12% of which being mDNS. Also, the number of "Wireless Event Too Big" logs had dropped to almost none after the change. This is a HUGE improvement and has stabilized the issue. I hope that this helps others down the road.

What was done:

Access-Lists (ACLs) were applied to each Aerohive switch in each building. The list contained several permit rules that allowed certain mDNS traffic to be accepted into the switch depending on the subnet it was originating from. Then at the end was a rule that blocked the rest of the mDNS traffic. The idea behind this is that since all of my switches connect directly to my core switch, allowing only certain mDNS traffic in depending on the subnet allows me to control and keep only the mDNS traffic within one building and not allowing it to traverse to others. I also did another ACL for IPv6 mDNS traffic and blocked that outright for everything.

Example ACL:

1. access-list 100 permit udp <subnet> any eq 5353
2. access-list 100 permit udp <subnet> any eq 5353
3. access-list 100 permit udp <subnet> any eq 5353
4. access-list 100 permit udp <subnet> any eq 5353
5. access-list 100 deny udp any any eq 5353
6. access-list 100 permit ip any any

Bonjour Gateway settings were completely redone. I only allowed specific services to be discovered from my Printer and IT-Devices VLANs and nothing else. Then I broke up each building with a unique bonjour gateway profile and applied Realms to them to keep only the devices needing to be discovered in each location.

Example Profile:

1. <specific services> From: Printer-VLAN To: Student + Faculty VLAN
2. <specific services> From: IT-Devices-VLAN To: Student + Faculty VLAN

Inter-station traffic was re-enabled to allow traffic to flow within the AP and not having to traverse the core switch, thus cutting down the amount of mDNS traffic going to the network.

I applied mDNS block rules on the IP Firewalls and applied these firewalls to certain User Profiles that do not need to be broadcasting their traffic to the network. This was then applied to the APs, which prevented some mDNS traffic from even entering the network.

Example Rule:

1. Deny From: any To: any UDP-5353

Then on my core switch, I applied a block for all mDNS traffic from going to my other campus, since there is no reason for them to need to access printers or AppleTVs on a campus that they are not even on.

Example ACL:

1. access-list 100 deny udp any any eq 5353
2. access-list 100 permit ip any any

And finally, I disabled Bonjour, AirPrint, Google Cloud Print, and mDNS on all of my copiers that were spitting out tons of queries and responses.

[cbuchenau]

Thanks a lot for sharing, that probably just made my day! (or probably more my week... ;-))

We are experiencing sudden and high mDNS bursts in a school environment right now, coming from Google Cast devices (Chromecast, TV, ...). Instead of 50 packets every 30 seconds, they suddenly flush out 20'000 packets in one burst.

I started to mitigate the behaviour by implementing simple ACLs on all Core/Distribution switches, blocking any mDNS traffic, which at least contains the bursts inside their building.

Carsten

[samantha_lynn]

I took a look at your case and I'm wondering if the CPU is always high, or if that was during a high utilization time. The tech data attached to the case shows CPU at around 20%, which isn't bad at all. I see we have case notes stating the CPU got up to 77%, but if that was during a high utilization time, then that wouldn't indicate a problem. The APs are designed to use their CPUs so anything below about 95% is not an issue. Could you tell me what the CPU is during a non-production time?

As far as inter-station traffic and limiting the mDNS traffic, you could try multicast to unicast conversion, but I would only suggest this if the CPU is low during a non-production time. If the CPU is still high during non-production hours, you might want to consider limiting multicast traffic at the switch level, but that could case some traffic to drop so I'd use that as a last resort.

Also, could you tell me what your subnet size is? Reducing the subnet, or introducing other VLANs to split the traffic up a bit, might also help.