Header Only - DO NOT REMOVE - Extreme Networks

Port Congestion


Userlevel 2
Hello Community,

I have been noted some congestion on ports connected to my GGC port channel and I can't figure out what is happening here.
Is there any issues about the flow control o buffer sharing?

The max shared buffer on ports are at 100% already and the version of the box (X460-24T) is 15.5.4.2 v1554b2

There's 4 sharings and all are droping packets.

Any thought about that?

Port Congestion Monitor Tue Jul 28 22:10:47 2015Port Link Packet
State Drop
================================================================================
1 A 40915183
2 A 41080619
3 A 40892745
4 A 40811054
5 A 39540703
6 A 39739627
7 A 39937877
8 A 40004508
9 A 40476716
10 A 40276213
11 A 40333055
12 A 40588116

Thanks in advance,

8 replies

Userlevel 6
Hello Julian This is very interesting as the switch will forward at wirespeed so if it is buffering then it means that the down stream switch may be telling it to do flow control and hold the packet. I am not aware of any buffering or flow issues but I would recommending going to the knowledge base and doing a search. The knowledge base has thoushands of already answered cases. If you put in port congestion you will get some articles that may help.

Here's the link https://gtacknowledge.extremenetworks.com/pkb_Home?q=port%20congestion&l=en_US

Thanks
P
Userlevel 6
Julian,

What is the utilization of these ports? (show port utilization)

Is the congestion counter continuously incrementing or only occasionally?

The port congestion counter is an egress counter. Normally this would increment when the traffic egressing this port exceeds the bandwidth of the link. So for example, if port 1 and 2 both send 1G of traffic to port 3 (and port 3 is also 1G) the port will not be able forward 2G of traffic so some traffic will be dropped (and the congestion counter will increment.)

If the congestion counters are continuously incrementing, it likely means the port is constantly over-utilized. A higher bandwidth link or additional LAG ports may be required.

If the counters occasionally increment it could be that you are seeing "micro-bursts." These are intermittent bursts of traffic that exceed the bandwidth and buffer capability of the port, but only for a very small amount of time (many times milliseconds or less.) Micro-bursts can happen on any link but they are more common as the average port utilization increases.

Of course it could also be pause frames as Paul stated above.
Userlevel 2
Hello Paul and Chad,

Thank you for your knowledge share about it.

First, I'll take a deep look in these knowledge base and see if there's something useful, by the way I see some information about diferente ASICS on the switch, is there possible on X460? How is that separated at the switch?

Second, look the utilization on the ports:

Port Link Link Rx Peak Rx Tx Peak Tx State Speed % bandwidth % bandwidth % bandwidth % bandwidth
================================================================================
1 A 1000 48.01 49.06 12.73 15.86
2 A 1000 49.84 55.02 12.26 12.85
3 A 1000 48.58 49.12 11.73 12.67
4 A 1000 50.36 52.06 13.43 14.89
5 A 1000 51.66 52.93 12.47 18.89
6 A 1000 52.61 55.84 13.29 14.94
7 A 1000 54.75 65.09 14.20 15.91
8 A 1000 52.69 57.52 12.54 19.52
9 A 1000 50.97 54.90 11.96 14.97
10 A 1000 52.50 52.50 11.24 14.76
11 A 1000 49.81 58.95 12.29 14.65
12 A 1000 49.84 59.35 11.82 15.76

And below are the two times output of the command " show port congestion"

Port Congestion Monitor Wed Jul 29 01:40:33 2015 Port Link Packet
State Drop
================================================================================
1 A 41967412
2 A 42189342
3 A 42037345
4 A 41910657
5 A 40616730
6 A 40832508
7 A 41012119
8 A 41119186
9 A 41587711
10 A 41377911
11 A 41428878
12 A 41693314

Port Congestion Monitor Wed Jul 29 01:40:41 2015 Port Link Packet
State Drop
================================================================================
1 A 41967917
2 A 42189527
3 A 42037345
4 A 41910931
5 A 40616756
6 A 40833393
7 A 41012498
8 A 41119494
9 A 41589635
10 A 41377973
11 A 41429601
12 A 41693680

So it's a constantly increasing. But as I said before the machine are of Google and there's no way to put more interfaces right now because of hardware limitation at Google side.

Thank you for your help!
Userlevel 5
Julian Eble wrote:

Hello Paul and Chad,

Thank you for your knowledge share about it.

First, I'll take a deep look in these knowledge base and see if there's something useful, by the way I see some information about diferente ASICS on the switch, is there possible on X460? How is that separated at the switch?

Second, look the utilization on the ports:

Port Link Link Rx Peak Rx Tx Peak Tx State Speed % bandwidth % bandwidth % bandwidth % bandwidth
================================================================================
1 A 1000 48.01 49.06 12.73 15.86
2 A 1000 49.84 55.02 12.26 12.85
3 A 1000 48.58 49.12 11.73 12.67
4 A 1000 50.36 52.06 13.43 14.89
5 A 1000 51.66 52.93 12.47 18.89
6 A 1000 52.61 55.84 13.29 14.94
7 A 1000 54.75 65.09 14.20 15.91
8 A 1000 52.69 57.52 12.54 19.52
9 A 1000 50.97 54.90 11.96 14.97
10 A 1000 52.50 52.50 11.24 14.76
11 A 1000 49.81 58.95 12.29 14.65
12 A 1000 49.84 59.35 11.82 15.76

And below are the two times output of the command " show port congestion"

Port Congestion Monitor Wed Jul 29 01:40:33 2015 Port Link Packet
State Drop
================================================================================
1 A 41967412
2 A 42189342
3 A 42037345
4 A 41910657
5 A 40616730
6 A 40832508
7 A 41012119
8 A 41119186
9 A 41587711
10 A 41377911
11 A 41428878
12 A 41693314

Port Congestion Monitor Wed Jul 29 01:40:41 2015 Port Link Packet
State Drop
================================================================================
1 A 41967917
2 A 42189527
3 A 42037345
4 A 41910931
5 A 40616756
6 A 40833393
7 A 41012498
8 A 41119494
9 A 41589635
10 A 41377973
11 A 41429601
12 A 41693680

So it's a constantly increasing. But as I said before the machine are of Google and there's no way to put more interfaces right now because of hardware limitation at Google side.

Thank you for your help!

Hi,

Here is one article which could guide you how identify "micro-burst" traffic.
https://gtacknowledge.extremenetworks.com/articles/How_To/How-to-identifying-microburst-congestion-w...

Best regards,
Userlevel 6
I think you are suffering from microbursts where the traffic shortly (a few milliseconds) exceed 100 % utilization. show port util cannot show microbursts and on high speed ports buffer quickly runs out. With all ports averaging above 50 % I bet you if you would take a look in real life those ports are on 100 % now and then.

One other thing, did you create all qosprofiles ? Every qosprofile you create takes away a bit of the shared buffer space from the ports. (check show port buffer).
https://gtacknowledge.extremenetworks.com/articles/Solution/Prevent-packet-drops
Userlevel 5
Julian,

Do you by chance have Aruba wireless in your environment? I would enable jumbo frames on all of the switches and see if that resolves your issues... "enable jumbo-frame port all" - Have seen in the field this occur and enabling jumbo frames has resolved it every time. There is no penalty to having jumbo-frame enabled.

Give it a shot...

Bill
Userlevel 2
Hello Bill,

Jumbo-frame is already on.
This switch is only for server communcation to our network, there's is a Google Global cache, Akamai and Netflix connected and then a port-channel of 20Gbps total (2x10Gbps) to a Juniper MX80.
We don't have Aruba at all, just Ubiquiti in some wireless clients.

And about the question of qos profile, only one configured as below:

SW-GGC # sh configuration | i qoscreate qosprofile "QP6"
configure dot1p type 7 qosprofile QP6
Userlevel 1
Julian,

in our environment we have a similar problem with IP Phones and it was due to CRC on stack cables between the stacked x460. Are you using standalone switch or stacked switchs ? If it is stacked, I suggest of watching the stack cables state with "show ports stack-port rxerror".

Tristan

Reply