topic Extreme Switch x670 bandwidth limited to 3g insted of 10g in ExtremeSwitching (EXOS/Switch Engine)

Extreme Switch x670 bandwidth limited to 3g insted of 10g

philippe_page — Thu, 22 Aug 2019 08:32:40 GMT

I would like to ask some question regarding our switch. I am testing the bandwidth of our switch. It is currently configured with 10g without any limits on its egress and ingress. However I am only able to get 3g bandwidth instead of 10g. I'll attach the port info detail here for further details:

Slot-1 switch # show port 1:33 info detail
Port: 1:33
Virtual-router: VR-Default
Type: SF+_UNKNOWN (Unsupported)
Random Early drop: Unsupported
Admin state: Enabled with 10G full-duplex
Link State: Active, 10Gbps, full-duplex
Link Ups: 3 Last: Tue Jun 11 21:04:52 2019
Link Downs: 2 Last: Tue Jun 11 21:02:55 2019

VLAN cfg:
Name: storage-net, 802.1Q Tag = 1006, MAC-limit = No-limit, Virtual router: VR-Default
Port-specific VLAN ID: 1006
Name: storage-net2, 802.1Q Tag = 2006, MAC-limit = No-limit, Virtual router: VR-Default
Port-specific VLAN ID: 2006
STP cfg:

Protocol:
Trunking: Master port with 2 members using algorithm address based - L3_L4
algorithm
Members: 1:33,2:33

EDP: Enabled

ELSM: Disabled
Ethernet OAM: Disabled
Learning: Enabled
Unicast Flooding: Enabled
Multicast Flooding: Enabled
Broadcast Flooding: Enabled
Jumbo: Disabled
Flow Control: Rx-Pause: Enabled Tx-Pause: Disabled
Priority Flow Control: Disabled
Reflective Relay: Disabled
Link up/down SNMP trap filter setting: Enabled
Egress Port Rate: No-limit
Broadcast Rate: No-limit
Multicast Rate: No-limit
Unknown Dest Mac Rate: No-limit
QoS Profile: None configured
Ingress Rate Shaping : Unsupported
Ingress IPTOS Examination: Disabled
Ingress 802.1p Examination: Enabled
Ingress 802.1p Inner Exam: Disabled
Ingress 802.1p Priority: 0
Egress IPTOS Replacement: Disabled
Egress 802.1p Replacement: Disabled
NetLogin: Disabled
NetLogin port mode: Port based VLANs
Smart redundancy: Enabled
Software redundant port: Disabled
IPFIX: Disabled Metering: Ingress, All Packets, All Traffic
IPv4 Flow Key Mask: SIP: 255.255.255.255 DIP: 255.255.255.255
IPv6 Flow Key Mask: SIP: ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff
DIP: ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff

Far-End-Fault-Indication: Disabled
Shared packet buffer: default
VMAN CEP egress filtering: Disabled
Isolation: Off
PTP Configured: Disabled
Time-Stamping Mode: None
Synchronous Ethernet: Unsupported
Dynamic VLAN Uplink: Disabled
VM Tracking Dynamic VLANs: Disabled

Re: Extreme Switch x670 bandwidth limited to 3g insted of 10g

Drew_C — Thu, 22 Aug 2019 08:34:27 GMT

What is your test method?

Re: Extreme Switch x670 bandwidth limited to 3g insted of 10g

philippe_page — Thu, 22 Aug 2019 08:40:26 GMT

Im using iperf3 on our servers connected to the switch. From one rack to the other. It is weird because when I am testing inter rack bandwidth it was able to show 10g but when I am testing on 1 rack it is limited to 3g only. I am looking for some difference in the switch configuration and I was able to find out that there are jumbo configured on the other switch. I was able to configure it also on the switch which was limiting me to only 3gb but I still get the same results.

Re: Extreme Switch x670 bandwidth limited to 3g insted of 10g

BradP — Thu, 22 Aug 2019 09:33:01 GMT

What version? We used to have a licence for 10G for unsupported optics, but that has since been removed in more recent versions of code

Re: Extreme Switch x670 bandwidth limited to 3g insted of 10g

philippe_page — Thu, 22 Aug 2019 09:39:28 GMT

# show version
Slot-1 : 800546-00-08 1628N-44360 Rev 8.0 BootROM: 1.0.2.1 IMG: 16.2.1.6
Slot-2 : 800546-00-08 1629N-48855 Rev 8.0 BootROM: 1.0.2.1 IMG: 16.2.1.6
Slot-3 :
Slot-4 :
Slot-5 :
Slot-6 :
Slot-7 :
Slot-8 :

Image : ExtremeXOS version 16.2.1.6 by release-manager
on Sat Aug 6 19:06:56 EDT 2016
BootROM : 1.0.2.1
Diagnostics : 5.4

Im using the same extreme switch for different racks. I still cant seem to identify what is causing this issue. I've look into auto negotiation and jumbo frames but still bandwidth is still limited to 3g instead of 10g

Re: Extreme Switch x670 bandwidth limited to 3g insted of 10g

philippe_page — Thu, 22 Aug 2019 09:43:01 GMT

Re: Extreme Switch x670 bandwidth limited to 3g insted of 10g

BradP — Thu, 22 Aug 2019 10:24:40 GMT

Do you have a lot of experience with iperf? I know there are ways to configure it to make sure it really tests throughput. Also, do other ports in your network perform at 10 with the same iperf setup?

Re: Extreme Switch x670 bandwidth limited to 3g insted of 10g

philippe_page — Thu, 22 Aug 2019 12:24:42 GMT

Other ports can perform 10g with the same setup, only on a specific rack which has not been able to. I've seen another difference and I'm not sure if this is relevant to what is causing this issue. Upon checking the "Type" of the cable it only shows "SF+_UNKNOWN (Unsupported)" but for the working switches it shows "SF+_CX1m (Unsupported)". I've tried removing and plugging back in the transceiver but it still shows unknown.

Re: Extreme Switch x670 bandwidth limited to 3g insted of 10g

EtherMAN — Thu, 22 Aug 2019 21:41:56 GMT

so what about things like flow control on or off ?
what does your interfaces on the switch show the bandwidth to be?
Are there any TX or RX errors on the switch port ?
Code revs the same?
Have you tried swapping the cables around and using one of the ten gig cables that get good throughput on the switch port that is having issues?

There can be big differences in the quality of some of these copper ten gig cables even form same manufacturer.

Re: Extreme Switch x670 bandwidth limited to 3g insted of 10g

philippe_page — Fri, 23 Aug 2019 06:03:43 GMT

flow control is off
it shows 10000 bandwidth on each interface
I'm not sure how to check TX or RX errors on switch ports. But I'll look into it.
I don't also know about code revs

I've also tried trying other cable which uses 10Gbps. However, it still shows "SF+_UNKNOWN (Unsupported)" and still limits the bandwidth to 3Gbpd.

Re: Extreme Switch x670 bandwidth limited to 3g insted of 10g

EtherMAN — Fri, 23 Aug 2019 22:06:27 GMT

Followup...
Show port xxx rx shows receive errors
Show port xxx Tx show tx errors
Show port xxx utilization will show close to real the usage the port is seeing.

If you look at the Utilization from the RX side where client is vs TX side where server is they should be equal.

If switch was problem then RX should be more than TX side showing the packet loss in the switch.

Iperf is still dependent on the cards and servers and how they perceive the connection between client and server and will attempt to match those conditions to optimize the settings for the nic cards ans data flows. If port settings are 100% the same and the server hardware/client hardware does not change then you should see some layer 1 errors somewhere in my mind. So what does the connect between racks look like?

There is a chance you could have a bad switch but I doubt it and am more inclined to believe there layer one issues or configuration differences that are being overlooked and not spotted..