topic 5520 EXOS wont link up to Cisco 3750x 802.11q trunk in ExtremeSwitching (EXOS/Switch Engine)

5520 EXOS wont link up to Cisco 3750x 802.11q trunk

Keith9 — Mon, 21 Dec 2020 22:17:26 GMT

I have a Cisco 3750x 2 switch stack on the second floor of one of our offices and we leased a small suite in the 1st floor of this building for training purposes. We had the cabling vendor run mm fiber between floor 1 and 2. For a few months the 1st floor was connected by 2 x 1 gbps fiber connections to the second floor using a Cisco 3650 switch. Two WAN connections come into the second floor cisco switch stack and use OSPF to get back to HQ. The EXOS switch on the 1st floor is just an access switch extension off of the 2nd floor. The config looks like this:

interface Port-channel1
description Trunk to 1566 2nd Floor
switchport trunk allowed vlan 1,172
switchport trunk encapsulation dot1q
switchport mode trunk
ip arp inspection trust
spanning-tree bpdufilter enable
ip dhcp snooping trust
!
interface GigabitEthernet1/1
description uplink to 1566 Fl2 Switch 1 Gi1/1/4
switchport trunk allowed vlan 1,172
switchport trunk encapsulation dot1q
switchport mode trunk
ip arp inspection trust
mls qos trust dscp
spanning-tree portfast edge
spanning-tree bpdufilter enable
channel-group 1 mode active
ip dhcp snooping trust
!
interface GigabitEthernet1/2
description uplink to 1566 Fl2 Switch 2 Gi1/1/4
switchport trunk allowed vlan 1,172
switchport trunk encapsulation dot1q
switchport mode trunk
ip arp inspection trust
mls qos trust dscp
spanning-tree portfast edge
spanning-tree bpdufilter enable
channel-group 1 mode active
ip dhcp snooping trust
!

Now that switch although it works, its quite a number of years old, so we are replacing it with a 5520-24W-EXOS running EXOS 31.1.1.3. I put the 4 port NIM in the right side of this switch and in the first two ports I populated them with SFP 1 Gbps 850 nm devices. I attempted to swing the uplinks to these ports with this configuration, and the ports show a 1000mbps link, there is NO traffic flow on it, and the 2nd floor switch shows its port-channel down.

enable diffserv examination port 33
disable dot1p examination port 33
configure ports 33 description-string "Uplink to 1566 2nd Floor"
configure ports 33 auto off speed 1000 duplex full
enable sharing 33 grouping 33-34 algorithm address-based L3_L4 lacp
configure vlan Default add ports 1-33,35-36 untagged
configure vlan VOICE add ports 1-33,35-36 tagged
configure qosprofile QP3 minbw 0 maxbw 100 ports 33
configure qosprofile QP5 minbw 0 maxbw 100 ports 33
disable edp ports 33
enable cdp ports 33
configure vlan default delete ports 34
enable diffserv examination port 34
disable dot1p examination port 34
configure ports 34 description-string "Uplink to 1566 2nd Floor"
configure ports 34 auto off speed 1000 duplex full
enable sharing 33 grouping 33-34 algorithm address-based L3_L4 lacp
disable edp ports 34
enable cdp ports 34

Even with just ONE of the fibers connected, though the port lights and the sh port command indicates a link… traffic will not flow between the EXOS and Cisco switches.

I never had this issue with 450G2’s or 690X EXOS switches, but in those cases I am either connecting two or more ports to a server, or in case of our core, the 690X’s I am doing MLAGs from all other switches in the HQ building. The 5520 is a new switch for us, and we ordered 14 of them… this is our one 24 port (the rest are 48 and have plans to outfit other buildings with them). All switches were ordered with the 4 port expantion on the side since this model does not have those ports by default like the previous generation.

The 2nd floor of this building will eventually get its two Cisco 3750X stacked switches replaced with 2 5520’s stacked and the optics switched out for 10 gig on both ends. It’s one gig now because thats all the cisco supports.

Any idea’s what I did incorrectly? This is fairly simple, vlan 1 is data, vlan 172 is voip. Its a small installation.

sh port 33 vid
Untagged
Port /Tagged VID(s)
-------- -------- ------------------------------------------------------------
33 Untagged 1
Tagged 172
sh port 34 vid
Untagged
Port /Tagged VID(s)
-------- -------- ------------------------------------------------------------
34 Untagged 1
Tagged 172
sh sharing
Load Sharing Monitor
Config Current Agg Min Ld Share Ld Share Agg Link Link Up
Master Master Control Active Algorithm Flags Group Mbr State Transitions
================================================================================
33 LACP 1 L3_L4 A 33 - R 7
L3_L4 34 - R 7
================================================================================

Re: 5520 EXOS wont link up to Cisco 3750x 802.11q trunk

Stefan_K_ — Mon, 21 Dec 2020 22:21:42 GMT

Greetings,

Is STP in use?

Can you provide the output of “show lacp counters”?

Best regards
Stefan

Re: 5520 EXOS wont link up to Cisco 3750x 802.11q trunk

Keith9 — Mon, 21 Dec 2020 22:29:29 GMT

I noticed above the default vlan was removed from port 34. I disabled sharing 33 so it would let me correct that.

When I readded sharing I thought, what other options do I have. Would this work you think?
enable sharing 33 grouping 33-34 algorithm port-based lacp

instead of

enable sharing 33 grouping 33-34 algorithm address-based L3_L4 lacp

I’m only using the address-based because that’s what I’ve used (and proven to work) on all of our other EXOS switches.

Re: 5520 EXOS wont link up to Cisco 3750x 802.11q trunk

Keith9 — Mon, 21 Dec 2020 22:33:29 GMT

STP is in use but should be disabled.

The counters are 0 probably because I just changed it.

Here is the config on the 2nd floor switch where this network originates in this building. I noticed bpdufilter is on the port channel but not the individual ports (is that wrong?). This links up no issues from Cisco to Cisco, just having difficulty from Cisco to EXOS.

interface Port-channel2
description Trunk to suite 112
switchport trunk allowed vlan 1,172
switchport trunk encapsulation dot1q
switchport mode trunk
ip arp inspection trust
spanning-tree bpdufilter enable
ip dhcp snooping trust
!

interface GigabitEthernet1/1/4
description Trunk to suite 112
switchport trunk allowed vlan 1,172
switchport trunk encapsulation dot1q
switchport mode trunk
ip arp inspection trust
mls qos trust dscp
spanning-tree portfast edge
channel-group 2 mode active
ip dhcp snooping trust
!

interface GigabitEthernet2/1/4
description Trunk to suite 112
switchport trunk allowed vlan 1,172
switchport trunk encapsulation dot1q
switchport mode trunk
ip arp inspection trust
mls qos trust dscp
spanning-tree portfast edge
channel-group 2 mode active
ip dhcp snooping trust
!

Re: 5520 EXOS wont link up to Cisco 3750x 802.11q trunk

Keith9 — Mon, 21 Dec 2020 22:42:42 GMT

EXOS STP configuration. since port 33 is in a sharing group I suppose that's why disable stpd s0 ports 33 does not also indicate port 34. Yes in fact that must be true because I tried to apply it and it says port 34 is not a member of STP domain s0, which makes sense because of treating that as one port with the sharing command.

sh configuration | i stpd
configure stpd s0 priority 0
enable stpd s0 auto-bind vlan VOICE
disable stpd s0 ports 33

sh stp
MSTP Global Configuration:
MSTP Region Name : 40882fe15400
MSTP Format Identifier : 0
MSTP Revision Level : 3
MSTP Digest : ac:36:17:7f:50:28:3c:d4:b8:38:21:d8:ab:26:de:62
Common and Internal Spanning Tree (CIST) : s0
Total Number of MST Instances (MSTI) : 0

Name Tag Flags Ports Bridge ID Designated Root Rt Port Rt Cost
s0 0000 EM---- 35 000040882fe15400 000040882fe15400 ------- 0

Total number of STPDs: 1 STP Flush Method: VLAN and Port
STP Filter Method: System-wide STP BPDU Forwarding: On
STP Multicast Send IGMP or MLD Query: On

Flags: (C) Topology Change, (D) Disable, (E) Enable, (R) Rapid Root Failover
(T) Topology Change Detected, (M) MSTP CIST, (I) MSTP MSTI

The only other concern I have is if that 4 port NIM in the side can accept 1GBPS SFP’s. It links, we get lights and we get the confirmation on the sh port 33-34 of a layer 2 link… just no traffic. They will be swapped out for 10 gig SFP+ optics once the switch on the 2nd floor is converted over. The 2nd floor switch is more complicated because its 96 ports, mostly used, trunks to a vmware, two wan links, OSPF, etc. It will take more downtime and planning, but it is on the 2021 project adjenda.

Re: 5520 EXOS wont link up to Cisco 3750x 802.11q trunk

Stefan_K_ — Mon, 21 Dec 2020 22:44:11 GMT

This is normal, since Port 33 is the load-sharing master and the VLAN config on this port applies on the whole LAG.

Well, it should match the configuration on the Cisco Switch. Talking about the cisco switch, is LACP activated there, since you are using LACP on the Extreme-Switch?
I don’t know cisco very well, but I think channel-group mode needs to be auto in order to use lacp and you need to specify a load-balance algorithm. (e.g. src-dst-ip).

I’m still missing show lacp counters?

Re: 5520 EXOS wont link up to Cisco 3750x 802.11q trunk

Keith9 — Mon, 21 Dec 2020 23:05:25 GMT

I think this is 0 because i disabled the sharing and re-enabled it. I’m not on-site to swing the cables and if I were I would have to wait until after 4:30 PM EST.

I think by default the cisco is

sh lacp counters

LACP PDUs dropped on non-LACP ports : 0
LACP Bulk checkpointed msgs sent : 0
LACP Bulk checkpointed msgs recv : 0
LACP PDUs checkpointed sent : 0
LACP PDUs checkpointed recv : 0

Lag Member Rx Rx Drop Rx Drop Rx Drop Tx Tx
Group Port Ok PDU Err Not Up Same MAC Sent Ok Xmit Err
--------------------------------------------------------------------------------
33 33 0 0 0 0 0 0
34 0 0 0 0 0 0

================================================================================

On the Cisco side right now here is the port-channel detail. You’ll notice the last time the port-channel bundled or un-bundled was last Friday late afternoon when I attempted to swing the cables from Cisco to EXOS.

sh etherchannel 2 detail
Group state = L2
Ports: 2 Maxports = 16
Port-channels: 1 Max Port-channels = 16
Protocol: LACP
Minimum Links: 0

Ports in the group:
-------------------
Port: Gi1/1/4
------------

Port state = Up Mstr Assoc In-Bndl
Channel group = 2 Mode = Active Gcchange = -
Port-channel = Po2 GC = - Pseudo port-channel = Po2
Port index = 0 Load = 0x00 Protocol = LACP

Flags: S - Device is sending Slow LACPDUs F - Device is sending fast LACPDUs.
A - Device is in active mode. P - Device is in passive mode.

Local information:
LACP port Admin Oper Port Port
Port Flags State Priority Key Key Number State
Gi1/1/4 SA bndl 32768 0x2 0x2 0x135 0x3D

Partner's information:

LACP port Admin Oper Port Port
Port Flags Priority Dev ID Age key Key Number State
Gi1/1/4 SA 32768 0007.7db6.e580 5s 0x0 0x1 0x11A 0x3D

Age of the port in the current state: 2d:16h:22m:28s

Port: Gi2/1/4
------------

Port state = Up Mstr Assoc In-Bndl
Channel group = 2 Mode = Active Gcchange = -
Port-channel = Po2 GC = - Pseudo port-channel = Po2
Port index = 0 Load = 0x00 Protocol = LACP

Flags: S - Device is sending Slow LACPDUs F - Device is sending fast LACPDUs.
A - Device is in active mode. P - Device is in passive mode.

Local information:
LACP port Admin Oper Port Port
Port Flags State Priority Key Key Number State
Gi2/1/4 SA bndl 32768 0x2 0x2 0x235 0x3D

Partner's information:

LACP port Admin Oper Port Port
Port Flags Priority Dev ID Age key Key Number State
Gi2/1/4 SA 32768 0007.7db6.e580 1s 0x0 0x1 0x11B 0x3D

Age of the port in the current state: 2d:16h:22m:21s

Port-channels in the group:
---------------------------

Port-channel: Po2 (Primary Aggregator)

------------

Age of the Port-channel = 102d:23h:36m:47s
Logical slot/port = 10/2 Number of ports = 2
HotStandBy port = null
Port state = Port-channel Ag-Inuse
Protocol = LACP
Port security = Disabled
Load share deferral = Disabled

Ports in the Port-channel:

Index Load Port EC state No of bits
------+------+------+------------------+-----------
0 00 Gi1/1/4 Active 0
0 00 Gi2/1/4 Active 0

Time since last port bundled: 2d:16h:22m:21s Gi2/1/4
Time since last port Un-bundled: 2d:16h:27m:53s Gi1/1/4

Re: 5520 EXOS wont link up to Cisco 3750x 802.11q trunk

Miguel-Angel_RO — Tue, 22 Dec 2020 01:46:51 GMT

Keith,

Here a working config I had between Cisco and Exos:

Cisco:

interface Ethernet1/17 description switch-Extreme switchport mode trunk switchport trunk native vlan 2 switchport trunk allowed vlan 2-1000 spanning-tree link-type point-to-point flowcontrol receive on flowcontrol send on channel-group 19 mode active no vtp

Exos (cluster):

enable sharing 16 grouping 16 algorithm address-based L2 lacp

create mlag peer "CORE_L3" configure mlag peer "CORE_L3" ipaddress x.y.z.w vr VR-Default enable mlag port 16 peer "CORE_L3" id 16

To be checked with your config

Mig

Re: 5520 EXOS wont link up to Cisco 3750x 802.11q trunk

Stefan_K_ — Tue, 22 Dec 2020 02:50:19 GMT

I think he won’t need the MLAG part… Just saying this so he isn’t confused. 🙂

Re: 5520 EXOS wont link up to Cisco 3750x 802.11q trunk

Keith9 — Wed, 23 Dec 2020 01:57:04 GMT

I put two new 5520’s in a stacked config in another office yesterday. I trunked two 1gbps copper ports from a Cisco 3750 switch stack to it for the transition period. IN Q1 we will transition devices to the EXOS platform, but the switch is online and up.

In this example it was easy…

on the EXOS side
enable sharing 1:1 grouping 1:1,2:1 algorithm address-based L3_L4 lacp

sh port 1:1,2:1 vid
Untagged
Port /Tagged VID(s)
-------- -------- ------------------------------------------------------------
1:1 Untagged 1
Tagged 172
2:1 Untagged 1
Tagged 172

On the Cisco Side

interface Port-channel2
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 1,172
switchport mode trunk
ip arp inspection trust
spanning-tree bpdufilter enable
ip dhcp snooping trust
!

interface GigabitEthernet1/0/20
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 1,172
switchport mode trunk
ip arp inspection trust
ip access-group acl1 in
mls qos trust dscp
spanning-tree portfast
channel-group 2 mode active
ip dhcp snooping trust
!

interface GigabitEthernet1/0/44
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 1,172
switchport mode trunk
ip arp inspection trust
ip access-group acl1 in
mls qos trust dscp
spanning-tree portfast
channel-group 2 mode active
ip dhcp snooping trust
!

The only difference is the Extreme side is a two switch stack and I’m using copper cables because that's all I have. Eventually, once all the site is moved over to it, the Cisco switches will go away there.

Don’t know why it doesn’t work at our small office training site. Can you use 1 GIG fiber SFP’s in the right 4 port module? I mean if you sh port it does indicate its linked up…

I have 10 GIG fiber SFP+ in the switch stack going to a new vmware ESXi server that will be phased in as well. That works as long as its not in sharing mode (which I think I have to figure out the vmware side of things for that).

Re: 5520 EXOS wont link up to Cisco 3750x 802.11q trunk

PeterK — Wed, 23 Dec 2020 03:42:15 GMT

I’m not so familiar with Cisco. But afaik the STP Default on Cisco switches is PVST+ and on newer EXOS it is MSTP. You should check Spanning Tree stats

Re: 5520 EXOS wont link up to Cisco 3750x 802.11q trunk

Miguel-Angel_RO — Wed, 23 Dec 2020 04:12:26 GMT

Keith,

The load-balancing algorithm must be the same on both sides.

Could you try with a L2 load-balancing? It is the default one usually.

on the EXOS side
enable sharing 1:1 grouping 1:1,2:1 algorithm address-based L2 lacp

Mig

Re: 5520 EXOS wont link up to Cisco 3750x 802.11q trunk

PeterK — Wed, 23 Dec 2020 04:35:30 GMT

Hi Mig,

sorry, but this is wrong. You can have different algorithms at both ends.

For Example: I’ve lots of installations where EXOS is connected to EOS S-Series.

EXOS hast L3_L4 and S-Series round-robin.

Re: 5520 EXOS wont link up to Cisco 3750x 802.11q trunk

Miguel-Angel_RO — Wed, 23 Dec 2020 04:58:51 GMT

Hi Peter,

You are right but this is leading to an unbalanced link load.

The purpose of this algorithm is explicitly to achieve a load-balancing.

Using different algorithms at both ends has in this perspective no meaning.

It is true it will work and with the current bandwidths on the ports it is no so often that we get load-balancing issues.

Concerning your specific issue, if you want to be sure about the role of STP in your trouble, you can just disable it on the ports of the LAG (if there are no risks of loop). This will exclude (or not) STP investigations.

Mig

Re: 5520 EXOS wont link up to Cisco 3750x 802.11q trunk

Keith9 — Wed, 23 Dec 2020 05:58:40 GMT

I’m not convinced that 1 gig sfp’s work properly. The same config I tried at this site worked fine in another site, from copper ports to copper ports, and 10 gig sfp+ ports to 10 gig sfp+ (populated with 850nm mm fiber) to a server.

I’ll be out of the office until January 4th and sometime in Q1 we will redo the switching in the second floor there and it will be from Extreme to Extreme, 10 gig SFP+ fiber optic modules.

Re: 5520 EXOS wont link up to Cisco 3750x 802.11q trunk

Adam_Minowski — Wed, 23 Dec 2020 23:41:01 GMT

In most cases HA is more important than LB. Of course proper balancing is important to have most of a “value” from multiple links, but LACP algorithm is irrelevant when LACP is not working at all 😉

About issue itself. Also I’d check LACP passive mode on Cisco switch, and first of all log files on both sides, plus LACP debug.

From Cisco documentation:

“Both the active and passive LACP modes enable ports to negotiate with partner ports to an EtherChannel based on criteria such as port speed, and for Layer 2 EtherChannels, based on trunk state and VLAN numbers. Ports can form an EtherChannel when they are in different LACP modes as long as the modes are compatible”.

As a closing note. If I understand correctly, similar configuation worked witch Cisco on older EXOS switches - in that case I’d suggest to contact GTAC.

Re: 5520 EXOS wont link up to Cisco 3750x 802.11q trunk

Adam_Minowski — Wed, 23 Dec 2020 23:45:54 GMT

If you use standalone ESXi you have to create static LAG on EXOS. ESXi do not support LACP. In order to have LACP based LAG you should have VMware Distributed Switch. Available with VWmare Enterprise Plus license or with NSX.

eg:

enable sharing 1 grouping 1,2 algorithm address-based L2

Mind no “lacp” statement at the end.

Re: 5520 EXOS wont link up to Cisco 3750x 802.11q trunk

Keith9 — Mon, 04 Jan 2021 22:35:26 GMT

eg:

enable sharing 1 grouping 1,2 algorithm address-based L2

Mind no “lacp” statement at the end.

Ah yes, in our HQ we have vcenter and therefore distributed switch capability, so that is setup. But at a branch office its just 1 1U server running free ESXi, so no need for vcenter when its just 1 system. But yeah its more about redundancy than load balancing. The small number of users at a branch office never need tons of bandwidth, just reliability.