7100/S-Series Outline of QSFP+ Port Expectations

FAQ_User — Thu, 28 Nov 2013 04:40:00 GMT

Article ID: 15133

Products
7100-Series, firmware 8.22.02.0012 and higher
S-Series, firmware 8.11.01.0014 and higher

Discussion
Quad Small Form-factor Pluggable (QSFP) feature support has been present for both 1x40Gb Ethernet and 4x10Gb Ethernet (a variation of QSFP+) since product release for the 7100-Series (though for best results please use firmware 7.91.03.0007 or higher) and since firmware 8.11.01.0014 release for the S-Series. This document is intended to provide an overview of relevant part numbers, functionality, and controls, as a starting point to either routinely working with this feature or attempting to troubleshoot it.

The part numbers which support QSFP+ functionality are:* 7100-Series *
71G11K2L2-48 40Gb/10Gb, 48 ports 100mb/1Gb SFP with 2 1/10Gb SFP+ and 2 40/10Gb QSFP ports
71G21K2L2-24P24 40Gb/10Gb, 24 ports 10/100/1000Mb RJ45 PoE+ & 24 ports 100Mb/1Gb SFP with 2 1/10Gb SFP+ and 2 40/10Gb QSFP ports
71G21K2L2-48P 40Gb/10Gb, 48 ports 10/100/1000Mb RJ45 PoE+ with 2 1/10Gb SFP+ and 2 40/10Gb QSFP ports
71K11L4-48 40Gb/10Gb, 48 ports 1/10Gb SFP+ with 4 40/10Gb QSFP ports
71K11L4-24 40Gb/10Gb, 24 ports 1/10Gb SFP+ with 4 40/10Gb QSFP ports
71K91L4-48 40Gb/10Gb, 48 ports 1/10GBASE-T with 4 40/10Gb QSFP ports
71K91L4-24 40Gb/10Gb, 24 ports 1/10GBASE-T with 4 10/40Gb QSFP ports

* S-Series *
SL8013-1206 40Gb/10Gb, 6 Ports 40GBASE-X via QSFP, VSB expansion slot
SL8013-1206-F8 40Gb/10Gb, 6 Ports 40GBASE-X via QSFP, 4 ports VSB via SFP+

* for both *
40GB-LR4-QSFP 40Gb/10Gb, 40GBASE-LR4 Transceiver, SM, 10Km, LC QSFP+
40GB-SR4-QSFP 40Gb/10Gb, 40GBASE-SR4 Transceiver, MM, 100m OM3 / 150m OM4, MPO QSFP+
40GB-C0.5-QSFP 40Gb/10Gb, Copper Direct Attach Cable, integrated transceivers, 1 QSFP+ to 1 QSFP+, 0.5m
40GB-C01-QSFP 40Gb/10Gb, Copper Direct Attach Cable, integrated transceivers, 1 QSFP+ to 1 QSFP+, 1m
40GB-C03-QSFP 40Gb/10Gb, Copper Direct Attach Cable, integrated transceivers, 1 QSFP+ to 1 QSFP+, 3m
40GB-C07-QSFP 40Gb/10Gb, Copper Direct Attach Cable, integrated transceivers, 1 QSFP+ to 1 QSFP+, 7m
40GB-F10-QSFP 40Gb/10Gb, Active Optical Direct Attach Cable, integrated transceivers, 1 QSFP+ to 1 QSFP+, 10m
40GB-F20-QSFP 40Gb/10Gb, Active Optical Direct Attach Cable, integrated transceivers, 1 QSFP+ to 1 QSFP+, 20m
10GB-4-C03-QSFP 10Gb only, Copper Direct Attach Hydra Cable, integrated transceivers, 1 QSFP+ to 4 SFP+, 3m
10GB-4-F10-QSFP 10Gb only, Active Optical Direct Attach Hydra Cable, integrated transceivers, 1 QSFP+ to 4 SFP+, 10m
10GB-4-F20-QSFP 10Gb only, Active Optical Direct Attach Hydra Cable, integrated transceivers, 1 QSFP+ to 4 SFP+, 20m
QSFP-SFPP-ADPT 10Gb only, Passive Optical Adapter, 1 integrated QSFP+ xceiver to 1 inserted (10Gb or 1Gb) SFP+ xceiverCompatibility:

Only Enterasys 40 Gigabit optical transceivers are supported in a QSFP+ port. Use of any other transceiver type will result in an error. Example message for an unauthenticated module in a QSFP+ port:
```
port fg.x.y contains an unauthenticated pluggable module(''/'')
```
Example port hold-down message for an unrecognized module in a QSFP+ port:
```
port fg.x.y will remain down because the pluggable module(''/'') is not supported
```
The 10GB-LRM-SFPP transceiver is not supported when plugged into a QSFP+ port via a QSFP-SFPP-ADPT. If an attempt is made to operate this device, the port will remain operationally down and a port hold-down error message (see above) will display.

Port modes, auto vs manual mode change:

A physical 40Gb QSFP+ port by default operates as one logical 40Gb "fg" port (1x40 mode), but may optionally be configured as four logical 10Gb "tg" ports (4x10 mode). The 10Gb QSFP+ to SFP+ "Hydra" cables (10GB-4-C03-QSFP, 10GB-4-F10-QSFP, and 10GB-4-F20-QSFP) provide for a single QSFP+ to four SFP+ cable conversion, effectively utilizing all four logical 10-gig ports available in 4x10 mode. Otherwise, a QSFP+ port in 4x10 mode operates as 10Gb instance 1 of 4. That arrangement generally requires 'set lacp enable', or STP will block 3 of the 4 logical ports - perhaps including the one logical port in actual use.

Out of the box or after a non-volatile memory clear, the fg ports are set to 1x40Gb mode.
After that, the ports remain in the same mode that was last used - unless a different mode is required by the inserted optic or by manual configuration.

Settings are largely automatic based upon inserted optic, with the following messaging possible:

code:

port fg.x.y contains a 4x10GB MAU but is currently in 40GB mode and will remain down until module x is reset

code:

port tg.x.y contains a 40GB MAU but is currently in 4x10Gb mode and will remain down until system is reset

The exception to the automatic treatment applies to these fiber cables:
The 10GB-4-F10-QSFP and 10GB-4-F20-QSFP require a manual port reset to 10000 Mbps (if not already at that setting), for proper operation.
The 40GB-F10-QSFP and 40GB-F20-QSFP require a manual port reset to 40000 Mbps (if not already at that setting), for proper operation.

To manually set the default 1x40GB mode to 4x10GB mode for one or more fg ports, enter the command
'set port speed <fg-port-string> 10000', specifying one or more of the 40GB ports.
If the targeted fg port(s) are already operating in the designated mode, then the operation is complete.
Otherwise, the targeted fg port(s) will move to a down state and remain down until the system is reset, issuing message:

code:

port fg.x.y is currently in 1x40Gb mode, the desired mode is 4x10Gb mode, the port will go down and remain down until module x is reset

After the system reset, each such port will come back up as four logical 10GB ports each using the port-string format tg.x.y.
Note: Prior to the system reset there is no direct command that will reverse this command. Instead, for this purpose physically (unplug and) plug in a 40Gb-auto-detecting QSFP (e.g. a 40GB-C03-QSFP), upon which the system will behave as if the 'set speed tg.x.y 40000' command had originally been entered.

To manually set the 4x10GB mode back to 1x40GB mode, enter the command
'set port speed <tg-port-string> 40000', specifying one or more of the 10GB ports.
If the targeted tg port(s) are already operating in the designated mode, then the operation is complete.
Otherwise, the targeted tg port groups (four logical tg ports per physical port) will move to a down state and remain down until the system is reset, issuing message:

code:

port tg.x.y is currently in 4x10Gb mode, the desired mode is 1x40Gb mode, the port will go down and remain down until the system is reset

After the system reset, each such port group will come back up as a single 40GB port using the port-string format fg.x.y.
Note: Prior to the system reset there is no direct command that will reverse this command. Instead, for this purpose physically (unplug and) plug in a 10Gb-auto-detecting QSFP (e.g. a 10GB-4-C03-QSFP), upon which the system will behave as if the 'set speed fg.x.y 10000' command had originally been entered.

Other:

Such messaging shows up in syslog, logging buffer, serial console session - and in a SSH/Telnet session if 'set logging here enable' is active. Only one port is cited in each message, though more ports (generally a total of 1 or 4 for the 7100, or 2 or 8 for the S) might be affected.
A 'clear config all' (or switch 7 toggle) sets all empty/unpopulated fg ports back to 1x40G mode.
Use the "forcelinkdown" feature (discussed in 5277) for best operation; to ensure that if one side of a connection is disabled, the other side reports link down.
7100K-Series fg ports each tie to an underlying chip and are thus mode-independent from each other, while 7100G-Series and S-Series fg ports are paired to each underlying chip, and are thus mode-synched with their adjacent port. For example, 'set port speed fg.1.1 10000' issued on a 7100K-Series would convert fg.1.1 to operate as four 10Gb ports, while if issued on a 7100G-Series or S-Series would convert fg.1.1-2 to operate as eight 10Gb ports; and so on.
The 40-gig fg port to 10-gig tg port conversion mapping formula is... fg.x.y <-> tg.x.(lowerports+(y*4)-3) through tg.x.(lowerports+(y*4)). For example, on a 71K11L4-24 which has 24 1/10Gb ports plus 4 40/10Gb ports, there are 24 lower-numbered ports occurring before the group of 40/10Gb ports. Thus, for this hardware... fg.2.1 converts to/from tg.2.(24+(1*4)-3) through tg.2.(24+(1*4)), or tg.2.25 through tg.2.28.
On the 7100G-Series, when the two QSFP+ ports are in 4x10 mode then the two SFP+ ports are no longer available (Oper Status shows as 'not-pres').
A 'show config' command by design does not give any indication that a QSFP port is in 4x10G mode.
What follows is how the port status might appear on an S-Series, in this case a SL8013-1206-F8 mapping fg.4.1-6 to tg.4.1-24 with mixed port modes in effect. It would be similar on a 7100-Series except that ports not currently present (due to their port mode) would be omitted rather than displayed as 'not-pres': S4 Chassis(su)->show port status *.4.* Port Alias Oper Admin Speed Duplex Type (truncated) Status Status (bps) ------------ ---------------- -------- ------- ------ ------- ------------------ tg.4.1 not-pres up unknown tg.4.2 not-pres up unknown tg.4.3 not-pres up unknown tg.4.4 not-pres up unknown tg.4.5 not-pres up unknown tg.4.6 not-pres up unknown tg.4.7 not-pres up unknown tg.4.8 not-pres up unknown tg.4.9 down up 10.0G full 4x10g-c03 other tg.4.10 down up 10.0G full 4x10g-c03 other tg.4.11 down up 10.0G full 4x10g-c03 other tg.4.12 down up 10.0G full 10g-x other tg.4.13 up up 10.0G full 4x10g-f10 other tg.4.14 up up 10.0G full 4x10g-f10 other tg.4.15 up up 10.0G full 4x10g-f10 other tg.4.16 down up 10.0G full 10g-x other tg.4.17 not-pres up unknown tg.4.18 not-pres up unknown tg.4.19 not-pres up unknown tg.4.20 not-pres up unknown tg.4.21 not-pres up unknown tg.4.22 not-pres up unknown tg.4.23 not-pres up unknown tg.4.24 not-pres up unknown com.4.1 down up 9.6K rs232 usb fg.4.1 down up 40.0G full unknown fg.4.2 down up 40.0G full unknown fg.4.3 not-pres up unknown fg.4.4 not-pres up unknown fg.4.5 down up 40.0G full 40g-f10 other fg.4.6 down up 40.0G full unknown 31 of 31 ports displayed, 3 port(s) with oper status 'up' or 'dormant'.

topic 7100/S-Series Outline of QSFP+ Port Expectations in FAQs

7100/S-Series Outline of QSFP+ Port Expectations