EXOS Understanding Buffer Sizes

[Anonymous]

Have been trying to work out the buffering sizes in EXOS on various switches but for the purpose of this query I have been experimenting on an X350.

So when you run either of these commands:

Show port buffer
debug hal show platform packet-buffer ports

You get the following components for ports 1-24 and 25-50, which I have assumed are based on 2 x ASIC’s:

• Total Packet Buffer Size
• Total Shared Buffer Size
• Port x Max Shared Buffer Usage
• QPx: Reserved Buffer

The outputs of the buffer commands are given below based on QoS profiles 1 to 8 being configured, removing one, then running the command again.

The problem I am having is getting the maths to match on what you would expect, so my questions are:

1. Is the total packet buffer size of 786432 the total packet buffer for all the ports or is it per ASIC 1-24 and 25-50?

2. How is the shared buffer size calculated:

2a. You would expect this to be a value remaining minus each of the reserved buffer space allocated to each of the QoS profiles per port, per ASIC.

2b. Whenever a QoS profile is deleted the total shared buffer size increases by 83456 for each ASIC (1-24 and 25-50), which divided by 1920 bytes is equal to 43.5 (Which does not equal all 24 or 49 ports), or divided by 24 is equal to 3477 bytes or divided again which is equal to 1739 (which does not equal 1920 bytes) - if you catch my drift 🙂

3. The max shared buffer usage is the one thing that always balances and if left as default is always 25% of the total shared buffer.Appreciate my understanding of this might be completely wrong but hoping someone can straighten it out.

Packet Buffer Allocation for ports in range 1-24
Total Packet Buffer Size: 786432 bytes, Not Overcommitted
Total Shared Buffer Size: 127488
Port 1 Max Shared Buffer Usage: 31872 bytes
QP1: Reserved Buffer: 1920 bytes
QP2: Reserved Buffer: 1920 bytes
QP3: Reserved Buffer: 1920 bytes
QP4: Reserved Buffer: 1920 bytes
QP5: Reserved Buffer: 1920 bytes
QP6: Reserved Buffer: 1920 bytes
QP8: Reserved Buffer: 1920 bytes

Packet Buffer Allocation for ports in range 1:1-24
Total Packet Buffer Size: 786432 bytes, Not Overcommitted
Total Shared Buffer Size: 210944
Port 1 Max Shared Buffer Usage: 52736 bytes
QP1: Reserved Buffer: 1920 bytes
QP3: Reserved Buffer: 1920 bytes
QP4: Reserved Buffer: 1920 bytes
QP5: Reserved Buffer: 1920 bytes
QP6: Reserved Buffer: 1920 bytes
QP8: Reserved Buffer: 1920 bytes

Packet Buffer Allocation for ports in range 1-24
Total Packet Buffer Size: 786432 bytes, Not Overcommitted
Total Shared Buffer Size: 294400
Port 1 Max Shared Buffer Usage: 73600 bytes
QP1: Reserved Buffer: 1920 bytes
QP3: Reserved Buffer: 1920 bytes
QP5: Reserved Buffer: 1920 bytes
QP6: Reserved Buffer: 1920 bytes
QP8: Reserved Buffer: 1920 bytes

Packet Buffer Allocation for ports in range 1-24
Total Packet Buffer Size: 786432 bytes, Not Overcommitted
Total Shared Buffer Size: 377856
Port 1 Max Shared Buffer Usage: 94464 bytes
QP1: Reserved Buffer: 1920 bytes
QP3: Reserved Buffer: 1920 bytes
QP4: Reserved Buffer: 1920 bytes
QP8: Reserved Buffer: 1920 bytes

Packet Buffer Allocation for ports in range 1-24
Total Packet Buffer Size: 786432 bytes, Not Overcommitted
Total Shared Buffer Size: 461312
Port 1 Max Shared Buffer Usage: 115328 bytes
QP1: Reserved Buffer: 1920 bytes
QP4: Reserved Buffer: 1920 bytes
QP8: Reserved Buffer: 1920 bytes

Packet Buffer Allocation for ports in range 1-24
Total Packet Buffer Size: 786432 bytes, Not Overcommitted
Total Shared Buffer Size: 544768
Port 1 Max Shared Buffer Usage: 136192 bytes
QP1: Reserved Buffer: 1920 bytes
QP8: Reserved Buffer: 1920 bytes

Packet Buffer Allocation for ports in range 25-50
Total Packet Buffer Size: 786432 bytes, Not Overcommitted
Total Shared Buffer Size: 231424
Port 25 Max Shared Buffer Usage: 57856 bytes
QP1: Reserved Buffer: 1920 bytes
QP2: Reserved Buffer: 1920 bytes
QP3: Reserved Buffer: 1920 bytes
QP4: Reserved Buffer: 1920 bytes
QP5: Reserved Buffer: 1920 bytes
QP6: Reserved Buffer: 1920 bytes
QP8: Reserved Buffer: 1920 bytes

Packet Buffer Allocation for ports in range 25-50
Total Packet Buffer Size: 786432 bytes, Not Overcommitted
Total Shared Buffer Size: 300032
Port 25 Max Shared Buffer Usage: 75008 bytes
QP1: Reserved Buffer: 1920 bytes
QP3: Reserved Buffer: 1920 bytes
QP4: Reserved Buffer: 1920 bytes
QP5: Reserved Buffer: 1920 bytes
QP6: Reserved Buffer: 1920 bytes
QP8: Reserved Buffer: 1920 bytes

Packet Buffer Allocation for ports in range 25-50
Total Packet Buffer Size: 786432 bytes, Not Overcommitted
Total Shared Buffer Size: 368640
Port 25 Max Shared Buffer Usage: 92160 bytes
QP1: Reserved Buffer: 1920 bytes
QP3: Reserved Buffer: 1920 bytes
QP5: Reserved Buffer: 1920 bytes
QP6: Reserved Buffer: 1920 bytes
QP8: Reserved Buffer: 1920 bytes

Packet Buffer Allocation for ports in range 25-50
Total Packet Buffer Size: 786432 bytes, Not Overcommitted
Total Shared Buffer Size: 437248
Port 25 Max Shared Buffer Usage: 109312 bytes
QP1: Reserved Buffer: 1920 bytes
QP3: Reserved Buffer: 1920 bytes
QP4: Reserved Buffer: 1920 bytes
QP8: Reserved Buffer: 1920 bytes

Packet Buffer Allocation for ports in range 25-50
Total Packet Buffer Size: 786432 bytes, Not Overcommitted
Total Shared Buffer Size: 505856
Port 25 Max Shared Buffer Usage: 126464 bytes
QP1: Reserved Buffer: 1920 bytes
QP4: Reserved Buffer: 1920 bytes
QP8: Reserved Buffer: 1920 bytes

Packet Buffer Allocation for ports in range 25-50
Total Packet Buffer Size: 786432 bytes, Not Overcommitted
Total Shared Buffer Size: 574464
Port 25 Max Shared Buffer Usage: 143616 bytes
QP1: Reserved Buffer: 1920 bytes
QP8: Reserved Buffer: 1920 bytes

Many thanks in advance

[Anonymous]

Still at a little loss as to the exact difference between the two and the high level mechanics behind it. So I'll detail my understanding and please correct me where I'm wrong:

When running the 'show ports buffer' command I believe you get the same output on both old and new gen switches - which is perhaps masking my understanding of the differences between the two.

Older Gen

So the older gen uses a shared buffer, but the ports can only use a certain percentage of the total remaining amount (after QoS Profile allocation) which by default is 25%. You can configure this, but you wouldn't typically want to set a port to 100% as this port could then potentially use 100% of the remaining shared buffer, thus starving any other ports of any shared buffer. It then gets a little murky in my understanding because all none profiled traffic would drop into QP1 anyway, so would that mean that although one port could potentially use 100% of the shared buffer there is still some buffer allocated as defined by QP1, (QP1: Reserved Buffer: 1920 bytes)?

With regards to the amount of shared buffer available, this is calculated as the amount left over after allocating space to QoS Profiles for fabric, CPU and ports, per port, per queue?

New Gen

The newer gen, which I believe is based on Broadcom smart buffering as detailed here https://www.broadcom.com/collateral/etp/SBT-ETP100.pdf (Please correct me if I'm wrong), still uses a shared buffer but its implementation is different than that on the older gen's.

So on the new gen's there is the conception of 'adaptive threshold buffering'. Does this mean that in difference to the old gen there isn't the capping in percentage of the shared buffer, as the new gen simply makes an adaptive allocation?

On the new gen there still seems to be the fixed allocation of buffer to QoS Profiles, but is there also (as on the old gen) an undetermined amount of memory also allocated to the fabric, CPU and ports, or is this accrued in the adaptive total remaining shared buffer?

I'm probing for these questions, not only to firm my understanding but to practically use in a situation of port congestion. For example, you have ports that are showing congestion in small amounts over a long period of time but no evidence of port over utilisation - assumption is a microburst or packets being dropped for a none congestion related issue. The show port congestion command is related to packet drops on egress (I believe), so to me this could mean the possible over subscription / burst of the multiples switch ports funnelling packets to this single port? The congestion (packet drops) could be either at the port, fabric or CPU level.

So aside from determining where the congestion is happening its useful for me to generically understand how it works as a whole, then later I can put the concept into practice in understanding / fixing what’s really happening and finding out exactly where in the architecture the problem is.

Diagnosing where the congestion is probably another conversation and another thread, but I’m hoping a good understanding of the buffering is a good starting place.

The other questions I’m interested in also in how the buffering works, handled, on ingress as well as egress and what’s the exception is for the 440G2’s and x620’s mentioned in the last post.

Big question but thanks in advance for any more information.

[Stephane_Grosj1]

For recent gen platform (G2's and x770, at the exception of x440G2 and x620), the technology is totally different. The buffer is larger (and dependent on the HW) and automatically adapt to the needs. You basically have a shared buffer for all ports, and when one port requires buffer, a portion of it is dynamically allocated to it.

So you have adaptive buffer thresholds which change in real-time based on instantaneous buffer usage. As the overall buffer usage increases, each queue’s buffer threshold becomes lower. Likewise, as the overall buffer usage decreases, each queue’s buffer threshold becomes higher. This allows maximum bursting capability while still providing fairness.

[Anonymous]

Hi Stephane,

Thanks for detail, that makes sense.

Would you by any chance know the detail on the more recent gen platform, perhaps the X460G2 and how the buffering works and calculated / assigned to that?

Much appreciated in advance.

[Stephane_Grosj1]

This is platform dependent.

For this older generation of platform, you have a 0.75MB packet buffer, as you pointed out. This is indeed per group of 24 ports on that platform.

By default, a small dedicated portion of that buffer is allocated to each queue created on each front panel ports, plus a slightly larger dedicated portion on the fabric ports and CPU port for all 8 queues.

You can configure that amount of dedicated buffer with the CLI command:
config qosprofile maxbuffer <%> port…

The remaining portion of the buffer is shared among all the ports, and a port can consume up to a certain percent of that shared pool at any given time.

The default percent is 25% but that can be changed with the CLI command:
config port shared-packet-buffer

This logic depends on the buffer technology used with a given platform. Most of the more recent Gen of platform have a different buffer technology (and a larger buffer).