Question

5520 EXOS wont link up to Cisco 3750x 802.11q trunk

  • 21 December 2020
  • 17 replies
  • 94 views

Userlevel 1

I have a Cisco 3750x 2 switch stack on the second floor of one of our offices and we leased a small suite in the 1st floor of this building for training purposes.  We had the cabling vendor run mm fiber between floor 1 and 2.  For a few months the 1st floor was connected by 2 x 1 gbps fiber connections to the second floor using a Cisco 3650 switch.  Two WAN connections come into the second floor cisco switch stack and use OSPF to get back to HQ.  The EXOS switch on the 1st floor is just an access switch extension off of the 2nd floor.  The config looks like this:

interface Port-channel1
 description Trunk to 1566 2nd Floor
 switchport trunk allowed vlan 1,172
 switchport trunk encapsulation dot1q
 switchport mode trunk
 ip arp inspection trust
 spanning-tree bpdufilter enable
 ip dhcp snooping trust
!
interface GigabitEthernet1/1
 description uplink to 1566 Fl2 Switch 1 Gi1/1/4
 switchport trunk allowed vlan 1,172
 switchport trunk encapsulation dot1q
 switchport mode trunk
 ip arp inspection trust
 mls qos trust dscp
 spanning-tree portfast edge
 spanning-tree bpdufilter enable
 channel-group 1 mode active
 ip dhcp snooping trust
!
interface GigabitEthernet1/2
 description uplink to 1566 Fl2 Switch 2 Gi1/1/4
 switchport trunk allowed vlan 1,172
 switchport trunk encapsulation dot1q
 switchport mode trunk
 ip arp inspection trust
 mls qos trust dscp
 spanning-tree portfast edge
 spanning-tree bpdufilter enable
 channel-group 1 mode active
 ip dhcp snooping trust
!

 

Now that switch although it works, its quite a number of years old, so we are replacing it with a 5520-24W-EXOS running EXOS 31.1.1.3.  I put the 4 port NIM in the right side of this switch and in the first two ports I populated them with SFP 1 Gbps 850 nm devices.  I attempted to swing the uplinks to these ports with this configuration, and the ports show a 1000mbps link, there is NO traffic flow on it, and the 2nd floor switch shows its port-channel down.

enable diffserv examination port 33
disable dot1p examination port 33
configure ports 33 description-string "Uplink to 1566 2nd Floor"
configure ports 33 auto off speed 1000 duplex full
enable sharing 33 grouping 33-34 algorithm address-based L3_L4 lacp
configure vlan Default add ports 1-33,35-36 untagged
configure vlan VOICE add ports 1-33,35-36 tagged
configure qosprofile QP3 minbw 0 maxbw 100 ports 33
configure qosprofile QP5 minbw 0 maxbw 100 ports 33
disable edp ports 33
enable cdp ports 33
configure vlan default delete ports 34
enable diffserv examination port 34
disable dot1p examination port 34
configure ports 34 description-string "Uplink to 1566 2nd Floor"
configure ports 34 auto off speed 1000 duplex full
enable sharing 33 grouping 33-34 algorithm address-based L3_L4 lacp
disable edp ports 34
enable cdp ports 34
 

Even with just ONE of the fibers connected, though the port lights and the sh port command indicates a link… traffic will not flow between the EXOS and Cisco switches.

I never had this issue with 450G2’s or 690X EXOS switches, but in those cases I am either connecting two or more ports to a server, or in case of our core, the 690X’s I am doing MLAGs from all other switches in the HQ building.  The 5520 is a new switch for us, and we ordered 14 of them… this is our one 24 port (the rest are 48 and have plans to outfit other buildings with them).  All switches were ordered with the 4 port expantion on the side since this model does not have those ports by default like the previous generation.

The 2nd floor of this building will eventually get its two Cisco 3750X stacked switches replaced with 2 5520’s stacked and the optics switched out for 10 gig on both ends.  It’s one gig now because thats all the cisco supports.

Any idea’s what I did incorrectly?  This is fairly simple, vlan 1 is data, vlan 172 is voip.  Its a small installation.

 sh port 33 vid
         Untagged
Port     /Tagged   VID(s)
-------- --------  ------------------------------------------------------------
33       Untagged  1
         Tagged    172
sh port 34 vid
         Untagged
Port     /Tagged   VID(s)
-------- --------  ------------------------------------------------------------
34       Untagged  1
         Tagged    172
sh sharing
Load Sharing Monitor
Config    Current Agg     Min    Ld Share        Ld Share  Agg Link  Link Up
Master    Master  Control Active Algorithm Flags Group     Mbr State Transitions
================================================================================
    33            LACP       1    L3_L4     A     33        -     R       7
                                  L3_L4           34        -     R       7
================================================================================


17 replies

Userlevel 5

Greetings,

Is STP in use?

Can you provide the output of “show lacp counters”?

Best regards
Stefan

Userlevel 1

I noticed above the default vlan was removed from port 34.  I disabled sharing 33 so it would let me correct that.

 

When I readded sharing I thought, what other options do I have.  Would this work you think?
enable sharing 33 grouping 33-34 algorithm port-based lacp

 

instead of 

enable sharing 33 grouping 33-34 algorithm address-based L3_L4 lacp

 

I’m only using the address-based because that’s what I’ve used (and proven to work) on all of our other EXOS switches.  

Userlevel 1

Greetings,

Is STP in use?

Can you provide the output of “show lacp counters”?

Best regards
Stefan

STP is in use but should be disabled.

The counters are 0 probably because I just changed it.

 

Here is the config on the 2nd floor switch where this network originates in this building.  I noticed bpdufilter is on the port channel but not the individual ports (is that wrong?).  This links up no issues from Cisco to Cisco, just having difficulty from Cisco to EXOS.

interface Port-channel2
 description Trunk to suite 112
 switchport trunk allowed vlan 1,172
 switchport trunk encapsulation dot1q
 switchport mode trunk
 ip arp inspection trust
 spanning-tree bpdufilter enable
 ip dhcp snooping trust
!

interface GigabitEthernet1/1/4
 description Trunk to suite 112
 switchport trunk allowed vlan 1,172
 switchport trunk encapsulation dot1q
 switchport mode trunk
 ip arp inspection trust
 mls qos trust dscp
 spanning-tree portfast edge
 channel-group 2 mode active
 ip dhcp snooping trust
!

interface GigabitEthernet2/1/4
 description Trunk to suite 112
 switchport trunk allowed vlan 1,172
 switchport trunk encapsulation dot1q
 switchport mode trunk
 ip arp inspection trust
 mls qos trust dscp
 spanning-tree portfast edge
 channel-group 2 mode active
 ip dhcp snooping trust
!

Userlevel 1

EXOS STP configuration.  since port 33 is in a sharing group I suppose that's why disable stpd s0 ports 33 does not also indicate port 34.  Yes in fact that must be true because I tried to apply it and it says port 34 is not a member of STP domain s0, which makes sense because of treating that as one port with the sharing command.

sh configuration | i stpd
configure stpd s0 priority 0
enable stpd s0 auto-bind vlan VOICE
disable stpd s0 ports 33

sh stp
                MSTP Global Configuration:
MSTP Region Name        : 40882fe15400
MSTP Format Identifier  : 0
MSTP Revision Level     : 3
MSTP Digest             : ac:36:17:7f:50:28:3c:d4:b8:38:21:d8:ab:26:de:62
Common and Internal Spanning Tree (CIST)        : s0
Total Number of MST Instances (MSTI)            : 0

Name       Tag  Flags  Ports Bridge ID        Designated Root  Rt Port Rt Cost
s0         0000 EM----    35 000040882fe15400 000040882fe15400 -------       0

Total number of STPDs: 1                STP Flush Method: VLAN and Port
STP Filter Method: System-wide          STP BPDU Forwarding: On
STP Multicast Send IGMP or MLD Query: On

Flags: (C) Topology Change, (D) Disable, (E) Enable, (R) Rapid Root Failover
       (T) Topology Change Detected, (M) MSTP CIST, (I) MSTP MSTI
 

 

The only other concern I have is if that 4 port NIM in the side can accept 1GBPS SFP’s.  It links, we get lights and we get the confirmation on the sh port 33-34 of a layer 2 link… just no traffic.  They will be swapped out for 10 gig SFP+ optics once the switch on the 2nd floor is converted over.  The 2nd floor switch is more complicated because its 96 ports, mostly used, trunks to a vmware, two wan links, OSPF, etc.  It will take more downtime and planning, but it is on the 2021 project adjenda.

Userlevel 5

I noticed above the default vlan was removed from port 34.  I disabled sharing 33 so it would let me correct that.

This is normal, since Port 33 is the load-sharing master and the VLAN config on this port applies on the whole LAG. 

When I readded sharing I thought, what other options do I have.  Would this work you think?
enable sharing 33 grouping 33-34 algorithm port-based lacp

 

instead of 

enable sharing 33 grouping 33-34 algorithm address-based L3_L4 lacp

 

I’m only using the address-based because that’s what I’ve used (and proven to work) on all of our other EXOS switches.  

Well, it should match the configuration on the Cisco Switch. Talking about the cisco switch, is LACP activated there, since you are using LACP on the Extreme-Switch? 
I don’t know cisco very well, but I think channel-group mode needs to be auto in order to use lacp and you need to specify a load-balance algorithm. (e.g. src-dst-ip).

 

I’m still missing show lacp counters?

Userlevel 1

I think this is 0 because i disabled the sharing and re-enabled it.  I’m not on-site to swing the cables and if I were I would have to wait until after 4:30 PM EST.

 

I think by default the cisco is 

sh lacp counters

LACP PDUs dropped on non-LACP ports : 0
LACP Bulk checkpointed msgs sent    : 0
LACP Bulk checkpointed msgs recv    : 0
LACP PDUs checkpointed sent         : 0
LACP PDUs checkpointed recv         : 0

Lag        Member     Rx       Rx Drop  Rx Drop  Rx Drop  Tx       Tx
Group      Port       Ok       PDU Err  Not Up   Same MAC Sent Ok  Xmit Err
--------------------------------------------------------------------------------
33         33         0        0        0        0        0        0
           34         0        0        0        0        0        0

================================================================================

 

 

On the Cisco side right now here is the port-channel detail.  You’ll notice the last time the port-channel bundled or un-bundled was last Friday late afternoon when I attempted to swing the cables from Cisco to EXOS.  

 

sh etherchannel 2 detail
Group state = L2
Ports: 2   Maxports = 16
Port-channels: 1 Max Port-channels = 16
Protocol:   LACP
Minimum Links: 0


                Ports in the group:
                -------------------
Port: Gi1/1/4
------------

Port state    = Up Mstr Assoc In-Bndl
Channel group = 2           Mode = Active          Gcchange = -
Port-channel  = Po2         GC   =   -             Pseudo port-channel = Po2
Port index    = 0           Load = 0x00            Protocol =   LACP

Flags:  S - Device is sending Slow LACPDUs   F - Device is sending fast LACPDUs.
        A - Device is in active mode.        P - Device is in passive mode.

Local information:
                            LACP port     Admin     Oper    Port        Port
Port      Flags   State     Priority      Key       Key     Number      State
Gi1/1/4   SA      bndl      32768         0x2       0x2     0x135       0x3D

Partner's information:

                  LACP port                        Admin  Oper   Port    Port
Port      Flags   Priority  Dev ID          Age    key    Key    Number  State
Gi1/1/4   SA      32768     0007.7db6.e580   5s    0x0    0x1    0x11A   0x3D

Age of the port in the current state: 2d:16h:22m:28s

Port: Gi2/1/4
------------

Port state    = Up Mstr Assoc In-Bndl
Channel group = 2           Mode = Active          Gcchange = -
Port-channel  = Po2         GC   =   -             Pseudo port-channel = Po2
Port index    = 0           Load = 0x00            Protocol =   LACP

Flags:  S - Device is sending Slow LACPDUs   F - Device is sending fast LACPDUs.
        A - Device is in active mode.        P - Device is in passive mode.

Local information:
                            LACP port     Admin     Oper    Port        Port
Port      Flags   State     Priority      Key       Key     Number      State
Gi2/1/4   SA      bndl      32768         0x2       0x2     0x235       0x3D

Partner's information:

                  LACP port                        Admin  Oper   Port    Port
Port      Flags   Priority  Dev ID          Age    key    Key    Number  State
Gi2/1/4   SA      32768     0007.7db6.e580   1s    0x0    0x1    0x11B   0x3D

Age of the port in the current state: 2d:16h:22m:21s

                Port-channels in the group:
                ---------------------------

Port-channel: Po2    (Primary Aggregator)

------------

Age of the Port-channel   = 102d:23h:36m:47s
Logical slot/port   = 10/2          Number of ports = 2
HotStandBy port = null
Port state          = Port-channel Ag-Inuse
Protocol            =   LACP
Port security       = Disabled
Load share deferral = Disabled

Ports in the Port-channel:

Index   Load   Port     EC state        No of bits
------+------+------+------------------+-----------
  0     00     Gi1/1/4  Active             0
  0     00     Gi2/1/4  Active             0

Time since last port bundled:    2d:16h:22m:21s    Gi2/1/4
Time since last port Un-bundled: 2d:16h:27m:53s    Gi1/1/4
 

Userlevel 6
Badge +1

Keith,

Here a working config I had between Cisco and Exos:

Cisco:

interface Ethernet1/17

  description switch-Extreme

  switchport mode trunk

  switchport trunk native vlan 2

  switchport trunk allowed vlan 2-1000

  spanning-tree link-type point-to-point

  flowcontrol receive on

  flowcontrol send on

  channel-group 19 mode active

  no vtp

Exos (cluster):

enable sharing 16 grouping 16 algorithm address-based L2 lacp

create mlag peer "CORE_L3"

configure mlag peer "CORE_L3" ipaddress x.y.z.w vr VR-Default

enable mlag port 16 peer "CORE_L3" id 16

To be checked with your config

Mig

Userlevel 5

I think he won’t need the MLAG part… Just saying this so he isn’t confused. :) 

Userlevel 1

I put two new 5520’s in a stacked config in another office yesterday.  I trunked two 1gbps copper ports from a Cisco 3750 switch stack to it for the transition period.  IN Q1 we will transition devices to the EXOS platform, but the switch is online and up.

In this example it was easy…

on the EXOS side
enable sharing 1:1 grouping 1:1,2:1 algorithm address-based L3_L4 lacp

sh port 1:1,2:1 vid
         Untagged
Port     /Tagged   VID(s)
-------- --------  ------------------------------------------------------------
1:1      Untagged  1
         Tagged    172
2:1      Untagged  1
         Tagged    172
 

On the Cisco Side

interface Port-channel2
 switchport trunk encapsulation dot1q
 switchport trunk allowed vlan 1,172
 switchport mode trunk
 ip arp inspection trust
 spanning-tree bpdufilter enable
 ip dhcp snooping trust
!

interface GigabitEthernet1/0/20
 switchport trunk encapsulation dot1q
 switchport trunk allowed vlan 1,172
 switchport mode trunk
 ip arp inspection trust
 ip access-group acl1 in
 mls qos trust dscp
 spanning-tree portfast
 channel-group 2 mode active
 ip dhcp snooping trust
!

interface GigabitEthernet1/0/44
 switchport trunk encapsulation dot1q
 switchport trunk allowed vlan 1,172
 switchport mode trunk
 ip arp inspection trust
 ip access-group acl1 in
 mls qos trust dscp
 spanning-tree portfast
 channel-group 2 mode active
 ip dhcp snooping trust
!

 

 

The only difference is the Extreme side is a two switch stack and I’m using copper cables because that's all I have.  Eventually, once all the site is moved over to it, the Cisco switches will go away there.

Don’t know why it doesn’t work at our small office training site.  Can you use 1 GIG fiber SFP’s in the right 4 port module?  I mean if you sh port it does indicate its linked up…

I have 10 GIG fiber SFP+ in the switch stack going to a new vmware ESXi server that will be phased in as well.  That works as long as its not in sharing mode (which I think I have to figure out the vmware side of things for that).

Userlevel 4
Badge

I’m not so familiar with Cisco. But afaik the STP Default on Cisco switches is PVST+ and on newer EXOS it is MSTP. You should check Spanning Tree stats

Userlevel 6
Badge +1

Keith,

 

The load-balancing algorithm must be the same on both sides.

Could you try with a L2 load-balancing? It is the default one usually.

on the EXOS side
enable sharing 1:1 grouping 1:1,2:1 algorithm address-based L2 lacp

 

Mig

Userlevel 4
Badge

Keith,

 

The load-balancing algorithm must be the same on both sides.

Could you try with a L2 load-balancing? It is the default one usually.

on the EXOS side
enable sharing 1:1 grouping 1:1,2:1 algorithm address-based L2 lacp

 

Mig

Hi Mig,

sorry, but this is wrong. You can have different algorithms at both ends.

For Example: I’ve lots of installations where EXOS is connected to EOS S-Series.

EXOS hast L3_L4 and S-Series round-robin.

Userlevel 6
Badge +1

Hi Peter,

You are right but this is leading to an unbalanced link load.

The purpose of this algorithm is explicitly to achieve a load-balancing.

Using different algorithms at both ends has in this perspective no meaning.

It is true it will work and with the current bandwidths on the ports it is no so often that we get load-balancing issues.

 

Concerning your specific issue, if you want to be sure about the role of STP in your trouble, you can just disable it on the ports of the LAG (if there are no risks of loop). This will exclude (or not) STP investigations.

 

Mig

Userlevel 1

I’m not convinced that 1 gig sfp’s work properly.  The same config I tried at this site worked fine in another site, from copper ports to copper ports, and 10 gig sfp+ ports to 10 gig sfp+ (populated with 850nm mm fiber) to a server.

I’ll be out of the office until January 4th and sometime in Q1 we will redo the switching in the second floor there and it will be from Extreme to Extreme, 10 gig SFP+ fiber optic modules.

Userlevel 1

 

Hi Peter,

You are right but this is leading to an unbalanced link load.

The purpose of this algorithm is explicitly to achieve a load-balancing.

 

In most cases HA is more important than LB. Of course proper balancing is important to have most of a “value” from multiple links, but LACP algorithm is irrelevant when LACP is not working at all ;)

About issue itself. Also I’d check LACP passive mode on Cisco switch, and first of all log files on both sides, plus LACP debug.

From Cisco documentation:

“Both the active and passive LACP modes enable ports to negotiate with partner ports to an EtherChannel based on criteria such as port speed, and for Layer 2 EtherChannels, based on trunk state and VLAN numbers. Ports can form an EtherChannel when they are in different LACP modes as long as the modes are compatible”.

As a closing note. If I understand correctly, similar configuation worked witch Cisco on older EXOS switches - in that case I’d suggest to contact GTAC.

 

Userlevel 1

I have 10 GIG fiber SFP+ in the switch stack going to a new vmware ESXi server that will be phased in as well.  That works as long as its not in sharing mode (which I think I have to figure out the vmware side of things for that).

If you use standalone ESXi you have to create static LAG on EXOS. ESXi do not support LACP. In order to have LACP based LAG you should have VMware Distributed Switch. Available with VWmare Enterprise Plus license or with NSX.

eg: 

enable sharing 1 grouping 1,2 algorithm address-based L2

Mind no “lacp” statement at the end.

Userlevel 1

I have 10 GIG fiber SFP+ in the switch stack going to a new vmware ESXi server that will be phased in as well.  That works as long as its not in sharing mode (which I think I have to figure out the vmware side of things for that).

If you use standalone ESXi you have to create static LAG on EXOS. ESXi do not support LACP. In order to have LACP based LAG you should have VMware Distributed Switch. Available with VWmare Enterprise Plus license or with NSX.

eg: 

enable sharing 1 grouping 1,2 algorithm address-based L2

Mind no “lacp” statement at the end.


Ah yes, in our HQ we have vcenter and therefore distributed switch capability, so that is setup.  But at a branch office its just 1 1U server running free ESXi, so no need for vcenter when its just 1 system.  But yeah its more about redundancy than load balancing.  The small number of users at a branch office never need tons of bandwidth, just reliability.

 

 

Reply