Header Only - DO NOT REMOVE - Extreme Networks

High TX Pause Count on uplink


Userlevel 1
Have a few different buildings running B5G124 switches (Fw 06.81.04.0001) with Fiber uplinks to our core. For the past few months been receiving intermittent high ping response (100-300 ms) with no packet loss. Clients are now intermittently experiencing delays of data up to approx 20 seconds every few hours. Noticed a high TX Pause Count on the B5 switches Fiber uplink. Noticed an Extreme article which spoke about disabling flow control on these uplink ports with high TX pause count. Wondering if this is a good idea or any other suggestions to evaluate?

4 replies

Userlevel 7
Hi Mark,

I would disable flow control on the B5 (set flowcontrol disable).

One big problem of the B5's Ethernet Flow Control is stopping all the traffic on the link, instead of stopping one class of service only as in Data Center Bridging with priority-based flow control (PFC).

Ethernet Flow Control is not VLAN aware either (neither is PFC), thus it is generally not useful on layer 2 uplinks (trunks).

You should consider what is supposed to happen if a link is congested.
  1. If one link is congested, the switch drops the frames it cannot buffer or deliver.
  2. If one link is congested, the switch can send pause frames to stop incoming traffic. The upstream device then has the burden to either buffer or discard the frames. If sufficient buffer space is available on the upstream device, the frames will be delayed instead of dropped.
It depends on the type of traffic how long it may be delayed by buffering (queuing delay) without negative effect. A storage metro-cluster does not tolerate more then a few milliseconds of delay, VoIP is supposed to allow up to 150ms delay, a Windows TCP session will time-out after 9 seconds. Delaying data longer than the application allows is often worse than just discarding the data.

TCP reacts to lost packets by reducing the data rate. Thus dropping frames on a congested link results in the end systems adjusting the data rate to the network conditions. Some TCP implementations react similarly to increasing delay, e.g. the default TCP on current Windows systems. Others, most notably the default TCP of Linux, ignore the delay. Thus added delay without packet loss does not result in reduced packet rate, and the congestion persists.

A queuing delay in the order of 20 seconds is too high for LAN applications. A B5 is not able to buffer that much data (assuming 1G or even 100M link speeds), thus dropping on the B5 seems the better solution to me. Introducing 20s of queuing delay into a network looks like bufferbloat to me.

Thanks,
Erik
Userlevel 1
Understood. We disabled flow control on these B5 and will see if the issue persists. One of the Extreme articles I'm reading states to not only disable flow control but to "clear port advertise ge.x.x pause". Does this help to not advertise that the port is capable of pausing when the flow control is disabled?
Userlevel 7
Hi Mark,

It may help not to confuse the end system. I have seen many servers sending high numbers of pause frames due to a NIC driver bug. Disabling flow control on the switch lets the switch ignore pause frames. Disabling advertisement of pause frame capability should result in the system connected to the port not sending or expecting any pause frames.

I have seen lots of problems with servers or storage systems that did not implement Ethernet Flow Control correctly. Those problems could usually be solved by disabling flow control on the switch.

Erik
Userlevel 1
Much appreciated. I'll strongly consider adding this to our troubleshooting and best practices list. We disabled the flow control which helped reduce the lag (no more 100-300ms lag warnings) but now we're just seeing the packets being dropped for about 20 seconds (which was suggested by Erik) and appears to be the best solution at the moment. The buffer overrun was only occurring on the upstream links from the edge to the core switch. The only recent significant change was an upgrade in bandwidth for the WAN link. Still searching for the cause.

Reply