Data Center (VDX)

 VDX port-channel and QoS

Jump to Best Answer
kyungchan tae's profile image
kyungchan tae posted 08-11-2020 06:44

Hello, 

I have configured the cir-rate to 4000000000 bps and added this to the port-channel. (outbound)

As I checked Interface stats, the port-channel TX Mbits/Sec is 8G, and the two physical interfaces are 4G each.

port-channel protocol is vLAG.
Is it correct to be assigned to each physical interface when constructing a cir-rate in vLag?

 

I listed my configuration below.

--------------------------------------------------------------------

1. Configuration

ip access-list extended DMI_QoS
 seq 10 permit ip any any
!
class-map DMI_QoS
 match access-group DMI_QoS
!
policy-map DMI_QoS
 class DMI_QoS
  police cir 4000000000
!
interface Port-channel 34
 vlag ignore-split
 speed 40000
 service-policy in DMI_QoS
 service-policy out DMI_QoS
 description To_A2-Private-BB
 switchport
 switchport mode trunk
 switchport trunk allowed vlan add 884
 switchport trunk tag native-vlan
 bpdu-drop enable all
 spanning-tree shutdown
 no shutdown
!
DMI-J04-Agg-1# show port-channel sum
Flags:  D - Down                P - Up in port-channel (members)
        U - Up (port-channel)   * - Primary link in port-channel
        S - Switched
        M - Not in use. Min-links not met
===== =============== ========== ===============
Group Port-channel    Protocol   Member ports   
===== =============== ========== ===============
34    Po 34   (SU)    vLAG       Fo 1/0/34* (P)
                                 Fo 2/0/34 (P)

 

---------------------------------------------------------------------------

2. Interface Stats

DMI-J04-Agg-1# show int stats detail int for 1/0/34
 Interface FortyGigabitEthernet 1/0/34 statistics (ifindex 4496572608)
                                   RX                              TX
             Packets   1370691649661                    981336658925
               Bytes1524991468632748                1059188641765505
            Unicasts   1370679886980                    981330835439
          Multicasts         8453727                         3866220
          Broadcasts         3308959                         1957266
              Errors               0                               0
            Discards          257494                               0
            Overruns               0       Underruns               0
               Runts               0
             Jabbers               0
                 CRC               0
        64-byte pkts     35217295304
   Over 64-byte pkts    315759386904
  Over 127-byte pkts     16635043142
  Over 255-byte pkts     16543705056
  Over 511-byte pkts      6712835637
 Over 1023-byte pkts    134655136037
 Over 1518-byte pkts    845168247582
           Mbits/Sec     3409.026464                     3998.462396
          Packet/Sec          470671                          436433
           Line-rate           8.71%                          10.17%

DMI-J04-Agg-1# show int stats detail int for 2/0/34
 Interface FortyGigabitEthernet 2/0/34 statistics (ifindex 8791539904)
                                   RX                              TX
             Packets          291782                    136788046082
               Bytes        33753254                 164605538829245
            Unicasts          118363                    136787873661
          Multicasts          173419                          172421
          Broadcasts               0                               0
              Errors               0                               0
            Discards               0                               0
            Overruns               0       Underruns               0
               Runts               0
             Jabbers               0
                 CRC               0
        64-byte pkts           67794
   Over 64-byte pkts           50569
  Over 127-byte pkts          173419
  Over 255-byte pkts               0
  Over 511-byte pkts               0
 Over 1023-byte pkts               0
 Over 1518-byte pkts               0
           Mbits/Sec        0.000000                     4014.862272
          Packet/Sec               0                          330840
           Line-rate           0.00%                          10.17%

DMI-J04-Agg-1# show int stats detail int port 34   
 Interface Port-channel 34 statistics (ifindex 671088674)
                                   RX                              TX
             Packets   1370704340265                   1118150299673
               Bytes1524995191395681                1223832186925037
            Unicasts   1370692404236                   1118144303799
          Multicasts         8627047                         4038586
          Broadcasts         3308985                         1957288
              Errors               0                               0
            Discards          257494                               0
            Overruns               0       Underruns               0
               Runts               0
             Jabbers               0
                 CRC               0
        64-byte pkts     35217363116
   Over 64-byte pkts    315769953516
  Over 127-byte pkts     16635231057
  Over 255-byte pkts     16543730353
  Over 511-byte pkts      6712847918
 Over 1023-byte pkts    134655150594
 Over 1518-byte pkts    845170063712
           Mbits/Sec      140.565704                     8011.224900
          Packet/Sec          177950                          660577
           Line-rate           0.18%                          10.01%

 

many thanks.

Truyen Phan's profile image
Truyen Phan

Hi,

You can assign the cir-rate at the port-channel level. We’ve seen a problem with using the ‘permit any any’ statement not working with the cir-rate in an old case, but was not reproducible. 

As a workaround,  can you try this permit statement instead of the ‘permit any any’ and see if the cir-rate is applied?

 

ip access-list extended DMI_QoS

 seq 10 permit ip 0.0.0.0 255.255.255.255 0.0.0.0 255.255.255.255

 

kyungchan tae's profile image
kyungchan tae

Hi Truyen,

Thank you for your kind reply.

 

Does it mean that cir-rate is assigned to each physical interface due to the 'permit any any' configuration?

 

I modified the ACL and tested it, but the results are the same.

 

Is it normal for the cir-rate to be assigned to the physical interface when the port-channel protocol is vLag or LACP?

 

Thanks.

Truyen Phan's profile image
Truyen Phan

Does it mean that cir-rate is assigned to each physical interface due to the 'permit any any' configuration?

No, based on your results and what we’ve seen in another older case, the cir-rate was not being applied since ‘permit any any’ was used in your access list instead of specific IPs or subnets. This was a one time occurrence and was fixed with a re-configuration. This could be expected behavior. See below for more details. 

I modified the ACL and tested it, but the results are the same.

Which NOS version are you using? Is this production traffic or lab test? 

Is it normal for the cir-rate to be assigned to the physical interface when the port-channel protocol is vLag or LACP?

Yes, the cir-rate can be applied to vLAG or LACP. The cir-rate can only be applied at the port-channel if the physical interface is bound to a port-channel. 

Can you add another interface from one of the switch into the port-channel to make it 3 interfaces total, then check if the 3rd interface of the port-channel also has TX of 4G? 

For example, 

PO 34 configure to have the below interfaces:

  • 1/0/34
  • 1/0/35
  • 2/0/34

Send 12G of traffic to PO34. Then, check what is the TX for interface 1/0/35? 

The port-channel is a global configuration, but each RB handles the traffic locally. So, if each RB receives 4Gb of traffic on it’s local port, then it will send out 4Gb. With the above configuration and adding in a 3rd port, I would expect 1/0/34 and 1/0/35 to have TX 2 Gb each while 2/0/34 have TX of 4Gb. 

kyungchan tae's profile image
kyungchan tae

Our NOS versions are 6.0.2g and 6.0.2ha.

It is production traffic and has difficulty adding member port of port-channel.

 

  • I expected that if we apply cir-rate of 4000000000 bps (Up to 4000 Mbits/Sec) to port-channel, the TX,RX limit of port-channel is 4000 Mbits/Sec.
  • However, the TX,RX limit for this port-channel is 8000 Mbits/Sec.
  • If this is normal, we will apply a cir-rate of 2000000000 bps (Up to 2000 Mbits/Sec) to port-channel and expect a TX,RX limit of 4000 Mbits/Sec.

 

According to what you mentioned, there is no difference in the cir-rate limit depending on the protocol type of the port channel. (vLag, LACP)

 

For example, even if only port-channel protocol was changed to LACP in the configuration that I initially shared, the port-channel TX,RX limit would be 8000 Mbits/Sec just like vLag. (The TX,RX limit of the two physical interfaces is 4000 Mbits/Sec)

 

Is this understanding correct?

 

Thanks.

Truyen Phan's profile image
Truyen Phan
  • I expected that if we apply cir-rate of 4000000000 bps (Up to 4000 Mbits/Sec) to port-channel, the TX,RX limit of port-channel is 4000 Mbits/Sec.
  • However, the TX,RX limit for this port-channel is 8000 Mbits/Sec.

Not sure if my previous response was clear. As mentioned, the port-channel configuration is a global configuration that both RB 1 and RB 2 are part of. However, each RB handles it’s traffic locally when it’s received. 

For example, both RBs receives traffic, since each RB has a port which are part of PO 34, each will RX,TX max of 4000 Mbit/sec according to the cir-rate. If you have another port added which connect to RB1 or RB2, then it will be split among those ports which are locally on the same RB. 

 

  • If this is normal, we will apply a cir-rate of 2000000000 bps (Up to 2000 Mbits/Sec) to port-channel and expect a TX,RX limit of 4000 Mbits/Sec

If you are looking to limit only 2000 Mbits/Sec to each interface on PO 34 which only has 2 interfaces that are part of RB1 and RB2, then setting the cir-rate to 2000000000 bps is correct. 

According to what you mentioned, there is no difference in the cir-rate limit depending on the protocol type of the port channel. (vLag, LACP)

 

For example, even if only port-channel protocol was changed to LACP in the configuration that I initially shared, the port-channel TX,RX limit would be 8000 Mbits/Sec just like vLag. (The TX,RX limit of the two physical interfaces is 4000 Mbits/Sec)

Is this understanding correct?

vLAG just means that the port-channel is connected/configured to multiple (different) R-Bridges. In this case, RB 1 and RB 2. Logically, RB 1 and RB 2 form the vLAG which talks LACP protocol for the port-channel. So, technically, you are already configured for both as a vLAG to use LACP protocol. 

 

Hope that clears it up a bit. 

kyungchan tae's profile image
kyungchan tae

I'm sorry to confuse you.

I think my LACP explanation was insufficient.

 

I wanted to know if there was a difference in the interface to which Cir-rate is assigned under the two conditions below.

 

  1. Two port-channel members are configured in two R-brides. (Ex : Te 1/0/1, Te 2/0/1)
  2. Two port-channel members are configured in one R-bridge. (Ex : Te 1/0/1, 1/0/2)

 

Will both of these conditions result in the following results?

 

  • Port-channel 1 stats : TX,RX limit of 8000 Mbits/Sec
  • Two physical interfaces stats : TX,RX limit of 4000 Mbits/Sec

 

Thank you for your kind reply.

Truyen Phan's profile image
Truyen Phan

Yes, I believe there will be a difference in results:

  1. Two port-channel members are configured in two R-brides. (Ex : Te 1/0/1, Te 2/0/1)

This is the result you were seeing:

  • Port-channel 1 stats : TX,RX limit of 8000 Mbits/Sec
  • Two physical interfaces stats : TX,RX limit of 4000 Mbits/Sec
  1. Two port-channel members are configured in one R-bridge. (Ex : Te 1/0/1, 1/0/2)

For this one, I would expect this result:

  • Port-channel 1 stats : TX,RX limit of 4000 Mbits/Sec
  • Two physical interfaces stats : TX,RX limit of 2000 Mbits/Sec on each interface
kyungchan tae's profile image
kyungchan tae

Hi Truyen,

 

Your answer was very helpful.

 

Thanks.