EXOS Rate Limiting and Rate Shaping

  • 0
  • 1
  • Question
  • Updated 2 years ago
  • Answered

Converting QoS EOS understanding to EXOS, which is causing me a little confusion.....

My understanding between the difference of rate limiting and rate shaping is the former simply sets a red line, so anything above it can be dropped. Whereas the later is more smooth and buffered.

Could someone please give me an example configuration of the two in EXOS to help me understand this?

Also a practical use for each. So for example, I might use traffic shaping for voice or video, and rate limiting for say backup traffic. The reason is the voice and video wouldn't be so tolerant to packet drops then say backup traffic, so I would want to buffer voice where I can rather than drop - would that be an accurate analogy?

Many thanks in advance.

Photo of Martin Flammia

Martin Flammia

  • 5,724 Points 5k badge 2x thumb

Posted 2 years ago

  • 0
  • 1
Photo of Mike D

Mike D, Alum

  • 3,852 Points 3k badge 2x thumb
Hello Martin,

Since you mentioned practical application I'll take a swing here.  I agree with your picture in general terms.  clip traffic vs buffer and schedule for later.  

My understanding of practical application is limited but I expect the  see the two treatments used in different places.  When managing voice, I thought dropped traffic the superior method.  Voice app can typically tolerate drops (thus udp) but not so good at recovering from delay, jitter and overhead (exhibit "A", udp again) Yes priority access to exit but I've never like the idea of shaping voice.  

Take fcoe on the other hand.   you can introduce all the variance in delivery interval your buffers can handle and the application will make sense of it.  fcoe will thank you for delivering each and every packet since the transport doesnt have recovery mechanisms of its own to depend on.
Drop packets in this network on the other hand, fcoe becomes testy.  
I see traffic shaping working here.  Tons of buffers in use on the switch and an environment that wont tolerate drops.  fcoe may be the most extreme example seems easiest to help me make sense of some of these concepts. 

my qos world-view and understanding of problems introduced by buffers congestion changes weekly.  Mean time - I find even the confusing queue treatment threads very instructive.   

Best,
Mike
Photo of Erik Auerswald

Erik Auerswald, Embassador

  • 12,782 Points 10k badge 2x thumb
Hi Martin, Mike,

there is not much packet buffer available in EXOS switches, so rate shaping can only be very limited. That is good, since shaping creates more problems than it solves. If a voice packet arrives at the end system after the preceding packet has left the anti-jitter buffer, it is dropped. Thus there is no need to hold onto VoIP packets for more than a say 100ms. Anyway, EXOS switches do not provide that much port buffer if the port runs at 1G or higher.

Most applications deal one way or another with packet loss, but packets arriving late are more problematic. They are often dropped at the end point (no need for the already congested network to transport them), or are already re-send, with the duplicate dropped at the receiver (so why deliver it at all, the network is congested already). TCPs congestion control is often based on packet loss, late packet delivery results in too much traffic sent by TCP.

I have seen a vivid example of traffic shaping in a satellite network. The WAN optimizer delayed e.g. DNS conversations for 30 seconds, but delivered every packet. The client re-tried twice (three requests in total), 1 second apart. The server replied to all three requests, the answers were delivered in 1s intervals as well. But since the traffic shaping WAN optimizer added 30s of delay, the client had timed out already and dropped the incoming replies.

Rate limiting is done to prevent a strict priority queue from starving out other traffic. This should be done for e.g. voice traffic at the access ports, possibly at highly oversubscribed uplinks, and at the WAN edge.

Brittle protocols like FCoE might profit from a finely tuned QoS setup and big buffers, but the only way to get them to fly is a sufficiently over-provisioned network.

Automatic traffic shaping as done by short-term oversubscribed switch ports (2 times 1G sending to one 1G) utilizing the port buffers works fine (unless the buffers are bloated -- see http://www.bufferbloat.net/). Traffic shaping (or rather packet pacing) by using 100M access ports for video cameras instead of 1G can prevent micro bursts from overwhelming 1G uplinks. But pretending 1G speed to an end system and using traffic shaping to send the data more slowly, as done in some WAN optimizers, just does not work. The WAN link might be at 100% utilisation, but goodput tends towards zero.

I would recommend to not use explicit traffic shaping, only policing (limiting) on ingress. Rate limiting can be useful to stop non-responsive (UDP) flows from drowning out other traffic. It is a safety measure if strict priority queuing is used. Rate limiting flooded traffic can safe the network in case of a layer 2 loop. But an enterprise LAN usually does not need any QoS setup to work fine, even for voice. VoIP even works fine over the Internet, with no QoS at all, see Skype or generic SIP telephony providers.

Traditional AQM (e.g. WRED on EXOS) might work, but usually needs very fine tuning to actually help, and might make the performance problems worse. Modern self-tuning AQM methods like CoDel and PIE have not made it into switch ASICs yet, but might be found in WAN routers, where they help the most.

Practical use of rate limiting:
  • prevent priority traffic from starving out everything else
  • signal TCP that it uses the available bandwidth
  • prevent flooded traffic from saturating a network during a layer 2 loop condition
Practical use of traffic shaping:
  • none that I know
Practical use of packet pacing (e.g. 100M access port instead of 1G; or done by sender):
  • prevent "micro burst" problems, e.g. for video
  • mitigate "TCP incast" problems, e.g. with cluster applications
I hope this post helps a little to understand QoS, rate limiting, and traffic shaping in the real world.

Regards,
Erik
Photo of Drew C.

Drew C., Community Manager

  • 37,322 Points 20k badge 2x thumb
Great reply Erik!  Thanks!
Photo of Mike D

Mike D, Alum

  • 3,852 Points 3k badge 2x thumb
Great info, Thanks Erik!
Photo of Martin Flammia

Martin Flammia

  • 5,724 Points 5k badge 2x thumb
Thanks both for posting as this has helped a lot. Much appreciated.

Might think of some more questions later on but I have one to ask now....

My plan moving forward might be to use WRR, so my thoughts are that I would then have no use for rate limiting since I can simply order the weights accordingly in sharing bandwidth.

The intention would be to use Strict Priority on say QP8 (Control traffic)  and QP7 (reserved for stacking) and weight any other queues used, including QP1

Would that be a good practice?

Many thanks.
Photo of Erik Auerswald

Erik Auerswald, Embassador

  • 12,782 Points 10k badge 2x thumb
The idea is good, but I am not sure if mixed queuing strategies can be implemented in EXOS. WRR should should be fine, since it prevents starving of lower priority queues.
Photo of Martin Flammia

Martin Flammia

  • 5,724 Points 5k badge 2x thumb
Great. Thanks Erik.