Port Congestion

  • 0
  • 1
  • Problem
  • Updated 8 months ago
  • Solved
Hello Community,


I have been noted some congestion on ports connected to my GGC port channel and I can't figure out what is happening here.
Is there any issues about the flow control o buffer sharing?

The max shared buffer on ports are at 100% already and the version of the box (X460-24T) is 15.5.4.2 v1554b2

There's 4 sharings and all are droping packets.

Any thought about that?


Port Congestion Monitor                                Tue Jul 28 22:10:47 2015Port      Link      Packet
          State     Drop
================================================================================
1         A         40915183
2         A         41080619
3         A         40892745
4         A         40811054
5         A         39540703
6         A         39739627
7         A         39937877
8         A         40004508
9         A         40476716
10        A         40276213
11        A         40333055
12        A         40588116



Thanks in advance,
Photo of Julian Eble

Julian Eble

  • 1,394 Points 1k badge 2x thumb

Posted 3 years ago

  • 0
  • 1
Photo of Paul Russo

Paul Russo, Alum

  • 9,694 Points 5k badge 2x thumb
Hello Julian  This is very interesting as the switch will forward at wirespeed so if it is buffering then it means that the down stream switch may be telling it to do flow control and hold the packet.  I am not aware of any buffering or flow issues but I would recommending going to the knowledge base and doing a search.  The knowledge base has thoushands of already answered cases.  If you put in port congestion you will get some articles that may help.

Here's the link https://gtacknowledge.extremenetworks.com/pkb_Home?q=port%20congestion&l=en_US


Thanks
P
Photo of Chad Smith

Chad Smith, Alum

  • 5,660 Points 5k badge 2x thumb
Julian,

What is the utilization of these ports? (show port utilization)

Is the congestion counter continuously incrementing or only occasionally?

The port congestion counter is an egress counter.  Normally this would increment when the traffic egressing this port exceeds the bandwidth of the link.  So for example, if port 1 and 2 both send 1G of traffic to port 3 (and port 3 is also 1G) the port will not be able forward 2G of traffic so some traffic will be dropped (and the congestion counter will increment.)

If the congestion counters are continuously incrementing, it likely means the port is constantly over-utilized.  A higher bandwidth link or additional LAG ports may be required.  

If the counters occasionally increment it could be that you are seeing "micro-bursts."  These are intermittent bursts of traffic that exceed the bandwidth and buffer capability of the port, but only for a very small amount of time (many times milliseconds or less.)  Micro-bursts can happen on any link but they are more common as the average port utilization increases.

Of course it could also be pause frames as Paul stated above.
(Edited)
Photo of Julian Eble

Julian Eble

  • 1,394 Points 1k badge 2x thumb
Hello Paul and Chad,

Thank you for your knowledge share about it.

First, I'll take a deep look in these knowledge base and see if there's something useful, by the way I see some information about diferente ASICS on the switch, is there possible on X460? How is that separated at the switch?

Second, look the utilization on the ports:


Port     Link    Link   Rx             Peak Rx       Tx            Peak Tx         State   Speed  % bandwidth    % bandwidth   % bandwidth   % bandwidth

================================================================================
1         A       1000     48.01         49.06         12.73           15.86
2         A       1000     49.84         55.02         12.26           12.85
3         A       1000     48.58         49.12         11.73           12.67
4         A       1000     50.36         52.06         13.43           14.89
5         A       1000     51.66         52.93         12.47           18.89
6         A       1000     52.61         55.84         13.29           14.94
7         A       1000     54.75         65.09         14.20           15.91
8         A       1000     52.69         57.52         12.54           19.52
9         A       1000     50.97         54.90         11.96           14.97
10        A       1000     52.50         52.50         11.24           14.76
11        A       1000     49.81         58.95         12.29           14.65
12        A       1000     49.84         59.35         11.82           15.76


And below are the two times output of the command " show port congestion"



Port Congestion Monitor                                Wed Jul 29 01:40:33 2015 Port      Link      Packet

          State     Drop
================================================================================
1         A         41967412
2         A         42189342
3         A         42037345
4         A         41910657
5         A         40616730
6         A         40832508
7         A         41012119
8         A         41119186
9         A         41587711
10        A         41377911
11        A         41428878
12        A         41693314

Port Congestion Monitor                                Wed Jul 29 01:40:41 2015 Port      Link      Packet

          State     Drop
================================================================================
1         A         41967917
2         A         42189527
3         A         42037345
4         A         41910931
5         A         40616756
6         A         40833393
7         A         41012498
8         A         41119494
9         A         41589635
10        A         41377973
11        A         41429601
12        A         41693680


So it's a constantly increasing. But as I said before the machine are of Google and there's no way to put more interfaces right now because of hardware limitation at Google side.

Thank you for your help!
(Edited)
Photo of Bin

Bin, Employee

  • 5,374 Points 5k badge 2x thumb
Hi,

Here is one article which could guide you how identify "micro-burst" traffic.
https://gtacknowledge.extremenetworks.com/articles/How_To/How-to-identifying-microburst-congestion-w...

Best regards,
Photo of OscarK

OscarK, ESE

  • 7,912 Points 5k badge 2x thumb
I think you are suffering from microbursts where the traffic shortly (a few milliseconds) exceed 100 % utilization. show port util cannot show microbursts and on high speed ports buffer quickly runs out. With all ports averaging above 50 % I bet you if you would take a look in real life those ports are on 100 % now and then.

One other thing, did you create all qosprofiles ? Every qosprofile you create takes away a bit of the shared buffer space from the ports. (check show port buffer).
https://gtacknowledge.extremenetworks.com/articles/Solution/Prevent-packet-drops
(Edited)
Photo of Bill Stritzinger

Bill Stritzinger, Alum

  • 6,036 Points 5k badge 2x thumb
Julian, 

Do you by chance have Aruba wireless in your environment?  I would enable jumbo frames on all of the switches and see if that resolves your issues... "enable jumbo-frame port all"   - Have seen in the field this occur and enabling jumbo frames has resolved it every time.  There is no penalty to having jumbo-frame enabled.  

Give it a shot...

Bill
Photo of Julian Eble

Julian Eble

  • 1,394 Points 1k badge 2x thumb
Hello Bill,

Jumbo-frame is already on.
This switch is only for server communcation to our network, there's is a Google Global cache, Akamai and Netflix connected and then a port-channel of 20Gbps total (2x10Gbps) to a Juniper MX80.
We don't have Aruba at all, just Ubiquiti in some wireless clients.

And about the question of qos profile, only one configured as below:

SW-GGC # sh configuration | i qoscreate qosprofile "QP6"
configure dot1p type 7 qosprofile QP6
(Edited)
Photo of Fauriant Tristan

Fauriant Tristan

  • 384 Points 250 badge 2x thumb
Julian,

in our environment we have a similar problem with IP Phones and it was due to CRC on stack cables between the stacked x460. Are you using standalone switch or stacked switchs ? If it is stacked,  I suggest of watching the stack cables state with "show ports stack-port rxerror".

Tristan