How to correctly implement traffic shaping for Metro-Ethernet providers WAN links in X690 ?

[Keith9]

Please see the article here from CenturyLink: https://www.centurylink.com/business/help/network/overview-of-traffic-shaping.html

Many telco's with a Metro-ethernet product suggest applying traffic shaping on the egress interface to mitigate issues with the carrier's policer dropping packets and causing TCP to drop off (creating a sawtooth like bandwidth graph profile).

What is the best way to do this on an X690 core licenced switch? Assuming you have Provider A coming into one port with a committed rate of 60mbps, and on that network are 3 other branch offices, each with 20mbps connections.

Then you have Provider B, which is a cable provider offering Metro-e, which maybe does not have the same requirements as applying traffic shaping? Say this comes into another port with a subscribed rate of 200mbps, and in that providers private cloud you have 4 other branch offices in different areas all at their own rates - which add up to the 200 at the HQ.

Would it be creating a pol file in VI and using an access-list on the switchport where the upstream WAN provider is plugged into?

Thanks!

[Keith9]

Well the ISP only offers one copper gigabit Ethernet hand-off.

I have found with our cable operator's enterprise metro ethernet circuit, that rate limiting is not required. I can fully saturate the link with iPerf and get a consistent 43 mbps for example on a 40mbps line. The handoff is from a Ciena switch that they provide (fiber in, copper Ethernet out).

Telco's used to be more strict. Back when we had Verizon years ago, they required the end user policing the rates, or we would see that sawtooth bandwidth pattern as they would furiously drop packets over the CIR, and TCP windowing would reset and drop speed, slowly increasing until packets are dropped again from exceeding that CIR.

We don't have Verizon anymore, we have another Telco and we have been throttling that out of habit, but I have not seen any documentation to state that its required. I have 60mbps in to our HQ, and 20mbps at 3 other sites. We moved the routing from a Cisco 2901 at our HQ to a pair of X690s running the core license and we are using OSPF for our interior routing protocol. I've noticed that an iPerf in one direction was showing around 800mbps, and 20mbps in the other direction. So I removed the limiters at the remote branch office Cisco 2901 routers and now I'm getting GIG speeds on this telco. So the telco goofed in our error and we will continue to keep it wide open. I can hit two sites simultaneously around 430mbps using iPerf3, which adds up to around 860mbps which shows me its a gigabit fiber that they forgot or incorrectly policed. I have to wonder when they screwed this up because we wouldn't have known since we rate limited each sub interface on the CIsco 2901 to a service output policy that defined the rate.

Its too bad we can't simplify the ACL's and do a match any. Logically in my head it sounds like it should work, but I have the mind of a human, not the mind of a machine! I guess when the day comes that our current Telco WAN provider figures out and "fixes" the bandwidth, I'll have to just manually put each IP address for each site in its own if statement. Each site has two networks, a 10.x.x.x/16 data network and a 172.16.x.x/24 voice network. The other side is a Cisco 2901 which I would just then re-attach the existing service output policy on that WAN subinterface, so that takes care of throttling in the other direction coming back to HQ.

We do have one site that is a X450-G2 stack with advanced edge license, but it is on the Cable operators metro Ethernet circuit, which they do not require shaping. We do have a 2901 at that site as well but its only for LTE backup so its peering with the X450-G2 with a higher OSPF cost and our private LTE VPN cloud from a third party provider. We needed to put that in place because although we can create gre tunnels in XOS, you cannot do OSPF over a tunnel name (it asks for vlan name in the configure ospf command). I have put in a feature request to gtac for this functionality.

It all depends on where the branch office is located as to which providers are available in the area. Otherwise I prefer dealing with the Cable operator. Its a little less expensive and its dead simple to setup. All sites can see everything in our private "cloud" and we can configure our endpoints however we want. Enabling a routing protocol creates a full mesh topology here. With the Telco provider, the branch office can only "see" the HQ. So for one branch to talk to another (rare but possible), the traffic has to come back to HQ first.

[StephenW]

Keith,

Unfortunately you can't do a "match any" with multiple destination IPs. You will need a entry for each destination IP

With your question about rate limiting each location I would make a meter for each location and limit that way. If each site is connected to it's own port then you can do a simple port rate limit to each port going to the site.

Stephen

[EtherMAN]

Keith it will also depend on the direction of your traffic. If must be controlled by you before it hits your WAN links. Even if you are successful in controlling the outbound of what you are putting on the Wan links the buffing and queing is just moving off the WAN and onto your switches still adversely affecting your traffic. I see you are working to get to get flow information too which will help you ID things further. I am a huge believer in shaping as close to source as possible when it need to go across a much smaller WAN link.

Good hunting

[Keith9]

Ok well I already have a meter in place which is triggered by an ACL on that port. It is just looking at tcp ports for VMWare vSphere replication to throtlle it down to meter 60mbps. Here is what it looks like (note if you have a more efficient way of writing this let me know):

entry ruleVR44046src {
if match all{
protocol TCP;
source-port 44046;
}
then {
Qosprofile qp1;
meter US60Mbps;
count US60Mbps-44046src-count;
}
}

entry ruleVR31031src {
if match all{
protocol TCP;
source-port 31031;
}
then {
Qosprofile qp1;
meter US60Mbps;
count US60Mbps-31031src-count;
}
}

entry ruleVR44046dst {
if match all{
protocol TCP;
destination-port 44046;
}
then {
Qosprofile qp1;
meter US60Mbps;
count US60Mbps-44046dst-count;
}

entry ruleVR31031dst {
if match all{
protocol TCP;
destination-port 31031;
}
then {
Qosprofile qp1;
meter US60Mbps;
count US60Mbps-31031dst-count;
}
}


Here is what's tricky, This port is the HQ, so it has 190mbps rate to the ISP.

At the other ends are other offices with the following committed rates:
40mbps
40mbps
10mbps
100mbps

So how would I rate it to each location? I would have to edit that ACL to add traffic to those destination IP addresses right? Like create these meters:
US40Mbps
US10Mbps
US100mbps

Then after those if statements for the vmware replication (which only goes to the site with 100mbps), do something like this right?
entry rule40mbps {
if match any{
destination-ip 10.80.0.0/16;
destination-ip 172.16.80.0/24;
destination-ip 10.4.0.0/16;
destination-ip 172.16.4.0/24;
}
then {
meter US40Mbps;
count US40Mbps-dst-count;
}
}
entry rule10mbps {
if match any{
destination-ip 10.50.0.0/16;
destination-ip 172.16.50.0/24;
}
then {
meter US10Mbps;
count US10Mbps-dst-count;
}
}
entry rule100mbps {
if match any{
destination-ip 10.30.0.0/16;
destination-ip 172.16.30.0/24;
destination-ip 192.168.31.0/24;
}
then {
meter US100Mbps;
count US100Mbps-dst-count;
}
}

3 of the other sites are terminated by Cisco 29xx series routers, so its very easy to police that end, and they are already setup.

Can I do that in an ACL, do a match any instead of a match all, and then put multiple possible conditions, so if either one of those hit it will do the actions specified in the then block?

Our voice networks are all 172.16.x.x/24 at each office, so maybe I should split those out and also assign them a higher qosprofile, like qp5 or qp6?