topic RE: IPTV multicast dropping packets in ExtremeSwitching (EXOS/Switch Engine)

IPTV multicast dropping packets

welisson — Thu, 12 Jul 2018 21:52:00 GMT

Hi guys,I have been working with IPTV for one year, so, all switch involved in this topology are x460, in this network I have mpls and vr created, nonetheless the multicast traffic is going through a specific VR with pim enabled.Everything is working properly, where on the main switch I have 2x460 in stack and two 10g ports going to other switch and 6 gigabit port going to another x460, in both case I have LACP enabled.
So, the curious situation that sometimes I can see some channels get a little bit checkered for some seconds, once it means that some dropped packed happened, but it just happens over traffic that is going through gigabit ports.In order for trying to solve this issue, we changed the SFP, optical patch cords, added more giga ports, increase the buffer ports to 100%, and with these stuff we could improve the transmission but not eliminated it in definitive.what we could notice that, sometime is still being logged some drop packets in these gigabit interfaces, where is causing this bad experience.I have tried to do a lot of thing to finished this dropped packets off. I could decrease the number of this incidents but not eliminated.I've been thinking that it could be the cpu process when using a gigaport where these are shared with the main cpu, different of 10g modules where they have a specific module to process all over the traffic.
Would someone get the same experience using x460 (not G2)? how would you solve this issue?
Tks

RE: IPTV multicast dropping packets

Karthik_Mohando — Fri, 13 Jul 2018 18:00:00 GMT

Welisson,

If the issue happens in many ports then i would suggest to disable the pause frames for the ports and check if that helps.

"disable flow-control rx-pause ports "

https://gtacknowledge.extremenetworks.com/articles/Q_A/Which-command-is-used-to-disable-rx-pause-fra...

RE: IPTV multicast dropping packets

welisson — Sat, 14 Jul 2018 01:20:00 GMT

Karthik, tks for answering
I did it but no successful, the same behavior.
Tks

RE: IPTV multicast dropping packets

welisson — Tue, 17 Jul 2018 18:35:00 GMT

Idk but I've been thinking that this switch is not being able to deal with this sort of traffic, with traffic mixed using gigabit interfaces.

RE: IPTV multicast dropping packets

Eric_Burke — Tue, 17 Jul 2018 18:50:00 GMT

Not sure if it would help (this is a bit out of left field). If the ports are SFP+, consider using a DAC (twinax) cable instead. My understanding is that the overhead on these ports is an order of magnitude less than fiber xcvr's. Not sure if that's possible in this series but might be worth a shot.

As a side note, here's the specs that make me suggest this. I'd (oddly enough) just read this yesterday and was surprised by the differences. ~~~~~~~~~~~~~~

Comparing Latency

The 10GBase-T PHY standard uses block coding to enable data to pass through the cable without error. The block encoding requires reading the data block into the transmitter PHY, running a mathematical function on the data, and sending the encoded data over the link. The opposite happens at the receiving end. The standard specifies 2.6 microseconds for the transmit-receive pair, and the size of the data block indicates that the wait time cannot be improved by approximately 2.0 microseconds per link. The SFP+ DAC cable uses simpler electronics without the need for code blocks. Typical latency is about 300 nanoseconds per link.

By comparison, we find that SFP+ DAC provides lower latency, but 10GBASE-T and SFP+ fiber cabling provides higher latency

RE: IPTV multicast dropping packets

welisson — Tue, 17 Jul 2018 19:09:00 GMT

Erik, tks.
So, in this case my matter is not being over 10Giga ports and so just 1gigabit ports.But about this article is quite interest to reading.
tks

RE: IPTV multicast dropping packets

Eric_Burke — Tue, 17 Jul 2018 19:09:00 GMT

I read that but it sounded like you were still using SFP. It's possible (someone here may know for sure) that a DAC would work in a 1GB SFP port. I too thought the numbers were interesting. That's a huge difference when you're down in the nanosecond range!

RE: IPTV multicast dropping packets

welisson — Tue, 17 Jul 2018 19:09:00 GMT

I'm supposing it wouldn't work for me, cause my distance between this both switch both "SiteA(main+stack 2x460) and siteB(just x460)" is around 5km, and this cable there is a limitation at most 10m.

RE: IPTV multicast dropping packets

Eric_Burke — Tue, 17 Jul 2018 19:20:00 GMT

For what it's worth, I think this site confirms the SFP+/DAC in an SFP port.

https://community.fs.com/blog/sfp-dac-twinax-cable-deploy-guide.html

Can I Use SFP+ DAC Twinax Cable in SFP Ports? Yes, SFP+ cables are backward compatible to SFP ports and will work fine. But SFP cables are not compatible to SFP+ ports. SFP cables can be plugged into SFP+ ports but they are not designed for 10Gb/s data rates.

RE: IPTV multicast dropping packets

EtherMan-Dupe — Tue, 17 Jul 2018 19:49:00 GMT

Lets not go down the path of speed of packets going across copper vs fiber or a DAC cable vs SFP fiber module... Both will deliver the same amount of packets... A few micro-seconds will not cause a stream of packets to drop frames. A buffer filling up due to micro bursts on your one gig interface could be your issue,,

RE: IPTV multicast dropping packets

Eric_Burke — Tue, 17 Jul 2018 19:49:00 GMT

Like I said, it was out of left field. Still interesting that there's an almost 7x increase in timing across a fiber module or BaseT port, even if it is minuscule. I'd not expected that.

RE: IPTV multicast dropping packets

welisson — Tue, 17 Jul 2018 19:49:00 GMT

it would be, however I have had increased the buffer to 100% as you can see below in all giga interface which belongs on this sharing/lacp

Packet Buffer Allocation for ports in range 1:1-30
Total Packet Buffer Size: 3145728 bytes, Not Overcommitted
Total Shared Buffer Size: 2934400
Port 1:24 Max Shared Buffer Usage: 1467136 bytes (50%)
QP1: Reserved Buffer: 1536 bytes
QP8: Reserved Buffer: 1536 bytes
Port 1:28 Max Shared Buffer Usage: 1467136 bytes (50%)
QP1: Reserved Buffer: 1536 bytes
QP8: Reserved Buffer: 1536 bytes
Packet Buffer Allocation for ports in range 2:1-30
Total Packet Buffer Size: 3145728 bytes, Not Overcommitted
Total Shared Buffer Size: 2934400
Port 2:24 Max Shared Buffer Usage: 1467136 bytes (50%)
QP1: Reserved Buffer: 1536 bytes
QP8: Reserved Buffer: 1536 bytes
Port 2:28 Max Shared Buffer Usage: 1467136 bytes (50%)
QP1: Reserved Buffer: 1536 bytes
QP8: Reserved Buffer: 1536 bytes

It is a default configuration
Port 2:27 Max Shared Buffer Usage: 586880 bytes (20%)
QP1: Reserved Buffer: 1536 bytes
QP8: Reserved Buffer: 1536 bytes

RE: IPTV multicast dropping packets

EtherMan-Dupe — Tue, 17 Jul 2018 20:32:00 GMT

Have you done a show congestion command on those one gig ports... This will show you if microbursts may be filling up the buffers and thus you are dropping packets. Having any mcast video stream going into buffers is a bad thing as the buffers cause the packets to arrive at edge out of order. Out of order packets in a constant mcast stream will be dropped ...

RE: IPTV multicast dropping packets

welisson — Tue, 17 Jul 2018 20:32:00 GMT

yep, I have. looking at the output below, we can see the packets dropped
Port Congestion Monitor
Port Link Packet
State Drop
================================================================================
1:24 A 20187
1:28 A 50409
2:24 A 17767
2:28 A 17588

this is the biggest problem, where once this drop starts happening automatically we can see some fail on the video stream.
" Having any mcast video stream going into buffers is a bad thing as the buffers cause the packets to arrive at edge out of order. Out of order packets in a constant mcast stream will be dropped ..."
What would you suggest with it?"

Tks

RE: IPTV multicast dropping packets

EtherMan-Dupe — Tue, 17 Jul 2018 23:26:00 GMT

Segment your network into smaller boundaries or get larger interfaces. Issue is that more than one gig of traffic is making it's way to those gig interfaces. That is why you have congestion and packet loss. Adding additional layer 3 subnets and segmenting your users into smaller chunks and groups or moving to ten gig interfaces... I have not had great luck in lagging one gig links together due to the fact a port lag is not an equal share between ports it bases it's sharing on what it sees at layer 3 and 4 if you have that option checked. So there are still chances you can have a single gig being overun in a lag group with the other port not even close to capacity.. We are seeing this at the ten gig level in our metro core which is a 40 gig lag... Moving all the lagged ports to 100 gig...

To prove this you should be able to do a filtered capture on a stream to a client and see the packets coming in out of order when the network is loaded..

RE: IPTV multicast dropping packets

welisson — Tue, 17 Jul 2018 23:26:00 GMT

Actually those ten giga interfaces aren't bound with giga lag, they are independent lag, and yes, I set up those lag in layer3 and all network is segmented where each service is using a specific vr, for instance VR Internet and VR MCAST(PIM).
i have been taking a look at some overload on those interfaces, but nothing.
Port Link Link Rx Peak Rx Tx Peak Tx
State Speed % bandwidth % bandwidth % bandwidth % bandwidth
================================================================================
1:24 A 1000 6.16 9.36 51.47 68.04
1:28 A 1000 5.89 7.42 55.08 67.73
2:24 A 1000 6.88 7.95 50.87 60.31
2:28 A 1000 5.13 6.01 47.10 55.74

This is a quite curious situation, I don't have any overload, CRC Error, nothing regarding QoS profile is set up, and even so, we have this drop, and the other question, it happens just over gigabit interfaces.

RE: IPTV multicast dropping packets

welisson — Fri, 10 Apr 2020 11:33:00 GMT

Updating this threat.

It really seems to be micro burst issue. I moved to 10g ports and the problem went out, but now. with peak around 8.5gbps the problem started happening again. the solution move to other switch/vendor, once that I don’t have more 10g port avaiable on this box.