topic VDX port-channel and QoS in Data Center (VDX)

VDX port-channel and QoS

kyungchan_tae — Tue, 11 Aug 2020 13:44:23 GMT

Hello,

I have configured the cir-rate to 4000000000 bps and added this to the port-channel. (outbound)

As I checked Interface stats, the port-channel TX Mbits/Sec is 8G, and the two physical interfaces are 4G each.

port-channel protocol is vLAG.
Is it correct to be assigned to each physical interface when constructing a cir-rate in vLag?

I listed my configuration below.

--------------------------------------------------------------------

1. Configuration

ip access-list extended DMI_QoS
seq 10 permit ip any any
!
class-map DMI_QoS
match access-group DMI_QoS
!
policy-map DMI_QoS
class DMI_QoS
police cir 4000000000
!
interface Port-channel 34
vlag ignore-split
speed 40000
service-policy in DMI_QoS
service-policy out DMI_QoS
description To_A2-Private-BB
switchport
switchport mode trunk
switchport trunk allowed vlan add 884
switchport trunk tag native-vlan
bpdu-drop enable all
spanning-tree shutdown
no shutdown
!
DMI-J04-Agg-1# show port-channel sum
Flags: D - Down P - Up in port-channel (members)
U - Up (port-channel) * - Primary link in port-channel
S - Switched
M - Not in use. Min-links not met
===== =============== ========== ===============
Group Port-channel Protocol Member ports
===== =============== ========== ===============
34 Po 34 (SU) vLAG Fo 1/0/34* (P)
Fo 2/0/34 (P)

---------------------------------------------------------------------------

2. Interface Stats

DMI-J04-Agg-1# show int stats detail int for 1/0/34
Interface FortyGigabitEthernet 1/0/34 statistics (ifindex 4496572608)
RX TX
Packets 1370691649661 981336658925
Bytes1524991468632748 1059188641765505
Unicasts 1370679886980 981330835439
Multicasts 8453727 3866220
Broadcasts 3308959 1957266
Errors 0 0
Discards 257494 0
Overruns 0 Underruns 0
Runts 0
Jabbers 0
CRC 0
64-byte pkts 35217295304
Over 64-byte pkts 315759386904
Over 127-byte pkts 16635043142
Over 255-byte pkts 16543705056
Over 511-byte pkts 6712835637
Over 1023-byte pkts 134655136037
Over 1518-byte pkts 845168247582
Mbits/Sec 3409.026464 3998.462396
Packet/Sec 470671 436433
Line-rate 8.71% 10.17%

DMI-J04-Agg-1# show int stats detail int for 2/0/34
Interface FortyGigabitEthernet 2/0/34 statistics (ifindex 8791539904)
RX TX
Packets 291782 136788046082
Bytes 33753254 164605538829245
Unicasts 118363 136787873661
Multicasts 173419 172421
Broadcasts 0 0
Errors 0 0
Discards 0 0
Overruns 0 Underruns 0
Runts 0
Jabbers 0
CRC 0
64-byte pkts 67794
Over 64-byte pkts 50569
Over 127-byte pkts 173419
Over 255-byte pkts 0
Over 511-byte pkts 0
Over 1023-byte pkts 0
Over 1518-byte pkts 0
Mbits/Sec 0.000000 4014.862272
Packet/Sec 0 330840
Line-rate 0.00% 10.17%

DMI-J04-Agg-1# show int stats detail int port 34
Interface Port-channel 34 statistics (ifindex 671088674)
RX TX
Packets 1370704340265 1118150299673
Bytes1524995191395681 1223832186925037
Unicasts 1370692404236 1118144303799
Multicasts 8627047 4038586
Broadcasts 3308985 1957288
Errors 0 0
Discards 257494 0
Overruns 0 Underruns 0
Runts 0
Jabbers 0
CRC 0
64-byte pkts 35217363116
Over 64-byte pkts 315769953516
Over 127-byte pkts 16635231057
Over 255-byte pkts 16543730353
Over 511-byte pkts 6712847918
Over 1023-byte pkts 134655150594
Over 1518-byte pkts 845170063712
Mbits/Sec 140.565704 8011.224900
Packet/Sec 177950 660577
Line-rate 0.18% 10.01%

many thanks.

Re: VDX port-channel and QoS

Truyen_Phan — Mon, 17 Aug 2020 11:40:59 GMT

Hi,

You can assign the cir-rate at the port-channel level. We’ve seen a problem with using the ‘permit any any’ statement not working with the cir-rate in an old case, but was not reproducible.

As a workaround, can you try this permit statement instead of the ‘permit any any’ and see if the cir-rate is applied?

ip access-list extended DMI_QoS seq 10 permit ip 0.0.0.0 255.255.255.255 0.0.0.0 255.255.255.255

Re: VDX port-channel and QoS

kyungchan_tae — Tue, 18 Aug 2020 10:46:00 GMT

Hi Truyen,

Thank you for your kind reply.

Does it mean that cir-rate is assigned to each physical interface due to the 'permit any any' configuration?

I modified the ACL and tested it, but the results are the same.

Is it normal for the cir-rate to be assigned to the physical interface when the port-channel protocol is vLag or LACP?

Thanks.

Re: VDX port-channel and QoS

Truyen_Phan — Tue, 18 Aug 2020 11:56:35 GMT

Does it mean that cir-rate is assigned to each physical interface due to the 'permit any any' configuration?

No, based on your results and what we’ve seen in another older case, the cir-rate was not being applied since ‘permit any any’ was used in your access list instead of specific IPs or subnets. This was a one time occurrence and was fixed with a re-configuration. This could be expected behavior. See below for more details.

I modified the ACL and tested it, but the results are the same.

Which NOS version are you using? Is this production traffic or lab test?

Is it normal for the cir-rate to be assigned to the physical interface when the port-channel protocol is vLag or LACP?

Yes, the cir-rate can be applied to vLAG or LACP. The cir-rate can only be applied at the port-channel if the physical interface is bound to a port-channel.

Can you add another interface from one of the switch into the port-channel to make it 3 interfaces total, then check if the 3rd interface of the port-channel also has TX of 4G?

For example,

PO 34 configure to have the below interfaces:

1/0/34
1/0/35
2/0/34

Send 12G of traffic to PO34. Then, check what is the TX for interface 1/0/35?

The port-channel is a global configuration, but each RB handles the traffic locally. So, if each RB receives 4Gb of traffic on it’s local port, then it will send out 4Gb. With the above configuration and adding in a 3rd port, I would expect 1/0/34 and 1/0/35 to have TX 2 Gb each while 2/0/34 have TX of 4Gb.

Re: VDX port-channel and QoS

kyungchan_tae — Tue, 18 Aug 2020 12:54:54 GMT

Our NOS versions are 6.0.2g and 6.0.2ha.

It is production traffic and has difficulty adding member port of port-channel.

I expected that if we apply cir-rate of 4000000000 bps (Up to 4000 Mbits/Sec) to port-channel, the TX,RX limit of port-channel is 4000 Mbits/Sec.

However, the TX,RX limit for this port-channel is 8000 Mbits/Sec.
If this is normal, we will apply a cir-rate of 2000000000 bps (Up to 2000 Mbits/Sec) to port-channel and expect a TX,RX limit of 4000 Mbits/Sec.

According to what you mentioned, there is no difference in the cir-rate limit depending on the protocol type of the port channel. (vLag, LACP)

For example, even if only port-channel protocol was changed to LACP in the configuration that I initially shared, the port-channel TX,RX limit would be 8000 Mbits/Sec just like vLag. (The TX,RX limit of the two physical interfaces is 4000 Mbits/Sec)

Is this understanding correct?

Thanks.

Re: VDX port-channel and QoS

Truyen_Phan — Tue, 18 Aug 2020 14:12:13 GMT

I expected that if we apply cir-rate of 4000000000 bps (Up to 4000 Mbits/Sec) to port-channel, the TX,RX limit of port-channel is 4000 Mbits/Sec.
However, the TX,RX limit for this port-channel is 8000 Mbits/Sec.

Not sure if my previous response was clear. As mentioned, the port-channel configuration is a global configuration that both RB 1 and RB 2 are part of. However, each RB handles it’s traffic locally when it’s received.

For example, both RBs receives traffic, since each RB has a port which are part of PO 34, each will RX,TX max of 4000 Mbit/sec according to the cir-rate. If you have another port added which connect to RB1 or RB2, then it will be split among those ports which are locally on the same RB.

If this is normal, we will apply a cir-rate of 2000000000 bps (Up to 2000 Mbits/Sec) to port-channel and expect a TX,RX limit of 4000 Mbits/Sec

If you are looking to limit only 2000 Mbits/Sec to each interface on PO 34 which only has 2 interfaces that are part of RB1 and RB2, then setting the cir-rate to 2000000000 bps is correct.

According to what you mentioned, there is no difference in the cir-rate limit depending on the protocol type of the port channel. (vLag, LACP)

For example, even if only port-channel protocol was changed to LACP in the configuration that I initially shared, the port-channel TX,RX limit would be 8000 Mbits/Sec just like vLag. (The TX,RX limit of the two physical interfaces is 4000 Mbits/Sec)
Is this understanding correct?

vLAG just means that the port-channel is connected/configured to multiple (different) R-Bridges. In this case, RB 1 and RB 2. Logically, RB 1 and RB 2 form the vLAG which talks LACP protocol for the port-channel. So, technically, you are already configured for both as a vLAG to use LACP protocol.

Hope that clears it up a bit.

Re: VDX port-channel and QoS

kyungchan_tae — Tue, 18 Aug 2020 15:25:32 GMT

I'm sorry to confuse you.

I think my LACP explanation was insufficient.

I wanted to know if there was a difference in the interface to which Cir-rate is assigned under the two conditions below.

Two port-channel members are configured in two R-brides. (Ex : Te 1/0/1, Te 2/0/1)
Two port-channel members are configured in one R-bridge. (Ex : Te 1/0/1, 1/0/2)

Will both of these conditions result in the following results?

Port-channel 1 stats : TX,RX limit of 8000 Mbits/Sec

Two physical interfaces stats : TX,RX limit of 4000 Mbits/Sec

Thank you for your kind reply.

Re: VDX port-channel and QoS

Truyen_Phan — Tue, 18 Aug 2020 15:36:40 GMT

Yes, I believe there will be a difference in results:

Two port-channel members are configured in two R-brides. (Ex : Te 1/0/1, Te 2/0/1)

This is the result you were seeing:

Port-channel 1 stats : TX,RX limit of 8000 Mbits/Sec

Two physical interfaces stats : TX,RX limit of 4000 Mbits/Sec

Two port-channel members are configured in one R-bridge. (Ex : Te 1/0/1, 1/0/2)

For this one, I would expect this result:

Port-channel 1 stats : TX,RX limit of 4000 Mbits/Sec

Two physical interfaces stats : TX,RX limit of 2000 Mbits/Sec on each interface

Re: VDX port-channel and QoS

kyungchan_tae — Tue, 18 Aug 2020 15:55:19 GMT

Hi Truyen,

Your answer was very helpful.

Thanks.