AV Network Issues: Flickering TVs and High Dante Audio Latency | Fabric engine & Voss | New Building

[Glenn_VUB]

Hey everyone,

We’re working on the network for a new building that has two routers on the ground floor (floor 0) and a bunch of switches spread across floors 0 to 5, all linked back to those two routers. Each floor has several switches—some general ones and some dedicated to AV (audio/video). If you check out the the topology, you’ll see what I mean. We have 4 normal switches & 4 AV switches that are setup the same way for each floor as this topology. On the right we have our 2Routers on floor 0.

We’ve been configuring a ton of devices on the AV switches, but we’ve run into a couple of problems:

First issue
On floor 1, there’s a room with multiple cameras and multiple TVs. The floor has four AV switches. If we connect the cameras to Switch 1 and the TVs to Switch 2, 3, or 4, the TVs flicker. We’re only seeing a few dropped packets, so it’s not obvious what’s causing it. Strangely, if we put everything on the same switch, the flickering stops.

Has anyone run into something like this before?

Second issue
We’re setting up some Dante-enabled devices for audio. Normally, if you speak into the mic, the audio should come out through the speakers, but we’re seeing high latency (50–70ms) on these devices, even though they’re all on the same local network. This might be why our speakers aren’t working correctly.

We suspect these two issues could be related, so we’d love any ideas on how to troubleshoot or fix them. We don’t have these problems in our older buildings, where we use 4950s in a stack with 7200VSPs. Our current firmware is 8.10.1+. Everything’s running Fabric Engine and VOSS on the routers.

For anyone not familiar with Dante, I’ve attached a PDF with more info.

 

Thanks in advance for any help!

– Glenn

[Glenn_VUB · Accepted solution]

Hello Yoann and anyone that might encounter this problem in the future.

We have solved the issues.

1. Issue where the audio was not going out of the speakers; this was related to the ip igmp querier, we changed this to something else and that fixed the audio and the dante devices connect and disconnect the whole time. The command under vlan config:
"ip igmp routed-spb-querier-addr [ip]"

2. We had latency, delay, and flickering TV screens. A colleague discovered that adjusting the FEC setting on our uplink ports was the solution. The default FEC, “cl108,” caused errors, so we switched to “cl74” on all uplinks, which resolved the problem.
For more information about FEC and configuring the FEC option:
Fabric Engine v9.1 User Guide |  https://documentation.extremenetworks.com/VOSS/SW/89/VOSSUserGuide/GUID-D841414F-6199-4BC8-92AF-43AD...
We are still investigating why the default FEC setting caused these issues and will open a case with Extreme Support on this.

 

[Yoann_Jonard]

If you have the support involved, it's better to wait for their feedback, I don't know nearly as much as they do.

For my understanding though, you have one L2 with multiple receivers and multiple senders within ? Or a L2 for cameras, and another for TV ?

Looking at the release notes, starting from 8.10.1 as you mentioned, there are few on multicast :

1. CFD-10189 (patched in 8.10.4.0)
   
2. CFD-10627 (patched in 8.10.4.0)
   
3. CFD-11866 (patched in 8.10.6.0)
   
4. CFD-11939 (patched in 8.10.6.0)
   

I don't have the big picture so I don't know if they are applicable to your design, but you can take a look anyway.

Regards

[Yoann_Jonard]

Hello Glen,

I would look into :

1. Are the ports QoS trusted ? By default I can see a port is L2 trusted (802.1p) and L3 trusted (DSCP), but if you are using auto-sense it's remapped.
2. Are you hitting a resources limit for multicast ? Check "show qos cosq-stats cpu-port" and "show qos cosq-stats int #INT-NAME" for any dropped packets.
   
   1. You can also take a look at the scaling for multicast for your switch model in the release notes here.
3. Are you using L2 multicast or L3 ?
4. Do you have rate limiting in place ? (show int gi rate-limit)
5. Do you have shaping in place ? (show int gi shape)

Best regards,

[Glenn_VUB]

Hello Yoann,

Thank you for your response!
QoS is trusted, no auto-sense on these ports, no drops in multicast queues, no rate limiting is hit either (or shaping).
We are using L2 multicast.
Yesterday, we had a call with Extreme's official support team and spent around 3-4 hours troubleshooting the issue. Unfortunately, we couldn't pinpoint the exact cause. Currently, we suspect it might be a Layer 1 (physical layer) problem. As a next step, we plan to replace SFPs, fibers, and other physical components to see if that resolves the issue.

If you have any other idea's on what to check. Feel free to guide me, ill keep you updated on the physical part.

Best regards,
Glenn