topic Re: 5520 switch, 1gbps optics in VIM, connected but no IP connectivity in ExtremeSwitching (EXOS/Switch Engine)

5520 switch, 1gbps optics in VIM, connected but no IP connectivity

Keith9 — Fri, 05 Mar 2021 23:35:42 GMT

I have a new 2 switch stack of 5520’s each with VIM-4X in the right side. I have 2 10gbps fibers that work perfectly in a load sharing to connect to another floor, and 2 Active Twinax to a vmware server in a load sharing link working perfectly.

I have a wireless rooftop backup antenna, its only backup so not mission critical as I have a 1gbps WAN and a 100mbps WAN already in this building, but I noticed no IP connectivity to this over the 1gbps fiber.

Port 1:58 shows 1000 FULL connection. The port link lights are up.
The port is in untagged vlan P2P-WIRELESS, and has an ip of 192.168.4.2/28.

The radio on the roof has an ip of 192.168.4.5/28 and is NOT pingable.

Neither is the other side of the link 192.168.4.1/28 at our HQ building about 500 feet away. No OSPF adjacency has formed becuase even though we have Layer 1 link, it doesn’t seem like traffic likes to pass on 1gbps SFP’s in VIM4X’s on 5520’s running EXOS 31.2.1.1.

sh iparp port 1:58 shows no mac entries, sh nodealias ports 1:58 shows nothing either.

I did disable port 1:58 then enable port 1:58, in the log it shows this…

03/05/2021 10:30:50.85 <Info:vlan.msgs.portLinkStateUp> Port 1:58 link UP at speed 1 Gbps and full-duplex

So link up but IP traffic won’t traverse.

sh port 1:58 vid
Untagged
Port /Tagged VID(s)
-------- -------- ------------------------------------------------------------
1:58 Untagged 44
vlan info

P2P-WIRELESS 44 192.168.4.2 /28 -f-------o---------------- ANY 1 /1 VR-Default

VLAN Interface with name P2P-WIRELESS created by user
Admin State: Enabled Tagging: 802.1Q Tag 44
Description: None
Virtual router: VR-Default
IP Anycast: Disabled
IPv4 Forwarding: Enabled
IPv4 MC Forwarding: Disabled
Primary IP: 192.168.4.2/28
IPv6 Forwarding: Disabled
IPv6 MC Forwarding: Disabled
IPv6: None
STPD: None
Protocol: Match all unfiltered protocols
Loopback: Disabled
NetLogin: Disabled
QosProfile: None configured
Egress Rate Limit Designated Port: None configured
Flood Rate Limit QosProfile: None configured
Suppress ARP: Disabled
Suppress ND: Disabled
Proxy ARP: Entry required
Ports: 1. (Number of active ports=1)
Untag: *1:58
Flags: (*) Active, (!) Disabled, (g) Load Sharing port
(b) Port blocked on the vlan, (m) Mac-Based port
(i) Port inactivated on the vlan due to VXLAN configuration
(a) Egress traffic allowed for NetLogin
(u) Egress traffic unallowed for NetLogin
(t) Translate VLAN tag for Private-VLAN
(s) Private-VLAN System Port, (L) Loopback port
(x) VMAN Tag Translated port
(A) Dynamically added by Auto-peering
(F) Dynamically added by Fabric Attach
(G) Multi-switch LAG Group port
(H) Dynamically added by MVRP
(I) Dynamically added by IDM
(N) Dynamically added by Netlogin
(U) Dynamically added uplink port
(V) Dynamically added by VM Tracking
pinging the device on the roof (which obvisouly has power since the laser is working in the sfp...) times out. Here’s the fiber signal levels for this 1gbps sfp:

sh port 1:58 transceiver information

Port Temp TxPower RxPower TxBiasCurrent Voltage-Aux1/ Voltage-Aux2
(Celsius) (dBm) (dBm) (mA) Vcc (Volts) (Volts)
================================================================================
1:58 27.83 -4.56 -14.24 2.63 3.37 N/A
================================================================================
N/A - parameter not applicable to transceiver connected to the port
D - Optic supports Digital Diagnostic Monitoring Interface (DDMI) which is off
* - value is out of normal range
-inf - negative infinity, parameter not defined

So how can one have layer 1 link but no IP connectivity?

The Cisco 3750X switch stack that was here previously had no problem with a Cisco SFP in sled Gi1/1/1.

Here is the previous Cisco config

interface GigabitEthernet1/1/1
description LightPointe FSO Link
switchport access vlan 44
ip arp inspection trust
ip access-group acl1 in
mls qos trust dscp
spanning-tree portfast edge
ip dhcp snooping trust

Here’s everything in EXOS having to do with port 1:58
sh configuration | i 1:58
configure vlan default delete ports 1:2,1:45,1:47,1:58,2:48,2:57,2:59
enable diffserv examination port 1:58
disable dot1p examination port 1:58
configure ports 1:58 description-string "P2P Wireless connection to HQ VL44"
configure ports 1:58 auto off speed 1000 duplex full
configure vlan P2P-WIRELESS add ports 1:58 untagged
configure qosprofile QP3 minbw 0 maxbw 100 ports 1:58
configure qosprofile QP5 minbw 0 maxbw 100 ports 1:58
disable edp ports 1:58
enable cdp ports 1:58
enable nodealias ports 1:58

And for the vlan

sh configuration | i P2P-WIRELESS
create vlan "P2P-WIRELESS"
configure vlan P2P-WIRELESS tag 44
configure vlan P2P-WIRELESS add ports 1:58 untagged
configure vlan P2P-WIRELESS ipaddress 192.168.4.2 255.255.255.240
enable ipforwarding vlan P2P-WIRELESS
enable bootprelay ipv4 vlan P2P-WIRELESS
configure ospf add vlan P2P-WIRELESS area 0.0.0.0
configure ospf vlan P2P-WIRELESS cost 15
configure ospf vlan P2P-WIRELESS priority 0
configure ospf vlan P2P-WIRELESS authentication encrypted md5 1 "encrypted string"

This should be pretty cut and dry, shouldnt it?

Re: 5520 switch, 1gbps optics in VIM, connected but no IP connectivity

Stefan_K_ — Sat, 06 Mar 2021 00:14:02 GMT

From where do you ping? Directly from the switch? (using the right VR?)

Do you see any arp-entries on this vlan?

show iparp vlan P2P-WIRELESS

Re: 5520 switch, 1gbps optics in VIM, connected but no IP connectivity

Keith9 — Sat, 06 Mar 2021 00:36:02 GMT

I ping from the switch itself.

just ping 192.168.4.5

or even

ping 192.168.4.5 from 192.168.4.2
Ping(ICMP) 192.168.4.5: 4 packets, 8 data bytes, interval 1 second(s).
Request timed out
Request timed out
Request timed out
44 bytes from 192.168.4.2: icmp_seq=0 Destination Host Unreachable
44 bytes from 192.168.4.2: icmp_seq=1 Destination Host Unreachable
44 bytes from 192.168.4.2: icmp_seq=2 Destination Host Unreachable
44 bytes from 192.168.4.2: icmp_seq=3 Destination Host Unreachable

sh iparp vlan P2P-WIRELESS
VR Destination Mac Age Static VLAN VID Port

Dynamic Entries : 129 Static Entries : 0
Pending Entries : 5

ARP address check:    Enabled              ARP refresh               :    Enabled
Timeout          :          20 minutes     ARP Sender-Mac Learning   :   Disabled
Locktime         :        1000 milliseconds
Retransmit Time :        1000 milliseconds
Reachable Time   :      900000 milliseconds (Auto)
Fast Convergence :         Off

ARP Global Settings
Max Entries         :    12288
Max Pending Entries :      256
Max Proxy Entries   :      256

On the other end at HQ, a EXOS X690 core I have no problem hitting the radio head to this exterier building.

That shows me what I know prior to last nights cutover from Cisco 3750X to Extreme 5520s at the remote building. The radio heads are working fine.

Because its goes to the roof, its fiber run. The fiber is powered by a POE injector plugged into a PDU that also runs to the roof with it. The fiber on the roof is into a 1000mbps 850nm mm optic. This optic works fine in X690’s, Cisco 3750, Extreme 450-G2’s but for some reason although it links up in a 5520, traffic cannot traverse it.

We call this vlan VL44 at the HQ.

Here’s the iparp from the core at the HQ building.

-Yellow-Core.3 # sh iparp vlan VL44
VR            Destination      Mac                Age Static VLAN          VID   Port
VR-Default    192.168.4.3      00:60:35:1f:9c:56    3      NO VL44          44    1:4
VR-Default    192.168.4.4      04:18:d6:58:c8:60   20      NO VL44          44    1:4
VR-Default    192.168.4.5      00:60:35:1f:9c:65    1      NO VL44          44    1:4
VR-Default    192.168.4.6      44:d9:e7:4a:6c:75   20      NO VL44          44    1:4
VR-Default    192.168.4.7      02:04:96:9d:52:ac   20      NO VL44          44    1:49

Dynamic Entries :         450             Static Entries            :          0
Pending Entries :          11
In Request       : 3698897465             In Response               :   23873222
Out Request      :   879593960             Out Response              :   92638294
Failed Requests :   143335743
Proxy Answered   :      835064
Rx Error         :     1705231             Dup IP Addr               :     192.168.4.1
Rejected Count   :     8104367             Rejected IP               : 169.254.252.38
Rejected Port    :        1:49             Rejected I/F              : Default

Max ARP entries :        8192             Max ARP pending entries   :        256
ARP address check:    Enabled              ARP refresh               :    Enabled
Timeout          :          20 minutes     ARP Sender-Mac Learning   :   Disabled
Locktime         :        1000 milliseconds
Retransmit Time :        1000 milliseconds
Reachable Time   :      900000 milliseconds (Auto)
Fast Convergence :         Off

Its not seing 192.168.4.2 at all.

sh vlan VL44
VLAN Interface with name VL44 created by user
    Admin State:         Enabled     Tagging:   802.1Q Tag 44
    Description:         HQ to 1566 Transport
    Virtual router:      VR-Default
    IPv4 Forwarding:     Enabled
    IPv4 MC Forwarding: Disabled
    Primary IP:          192.168.4.8/28
    IPv6 Forwarding:     Disabled
    IPv6 MC Forwarding: Disabled
    IPv6:                None
    STPD:                s0(Enabled,Auto-bind)
    Protocol:            Match all unfiltered protocols
    Loopback:            Disabled
    NetLogin:            Disabled
    OpenFlow:            Disabled
    QosProfile:          None configured
    Egress Rate Limit Designated Port: None configured
    Flood Rate Limit QosProfile:       None configured
    Suppress ARP:        Disabled
    Proxy ARP:           Entry required
    Ports:   2.           (Number of active ports=2)
       Untag:    *1:4
       Tag:    *1:49g
        Flags:    (*) Active, (!) Disabled, (g) Load Sharing port
                  (b) Port blocked on the vlan, (m) Mac-Based port
                  (i) Port inactivated on the vlan due to VXLAN configuration
                  (a) Egress traffic allowed for NetLogin
                  (u) Egress traffic unallowed for NetLogin
                  (t) Translate VLAN tag for Private-VLAN
                  (s) Private-VLAN System Port, (L) Loopback port
                  (x) VMAN Tag Translated port
                  (A) Dynamically added by Auto-peering
                  (F) Dynamically added by Fabric Attach
                  (G) Multi-switch LAG Group port
                  (H) Dynamically added by MVRP
                  (I) Dynamically added by IDM
                  (N) Dynamically added by Netlogin
                  (U) Dynamically added uplink port
                  (V) Dynamically added by VM Tracking

Yellow-Core.7 # sh port 1:4 transceiver information

Port      Temp    TxPower RxPower TxBiasCurrent Voltage-Aux1/ Voltage-Aux2
       (Celsius)   (dBm)    (dBm)     (mA)         Vcc (Volts)       (Volts)
================================================================================
1:4       28.84    -5.86    -7.05     2.64          3.36          N/A
================================================================================
      N/A - parameter not applicable to transceiver connected to the port
      *    - value is out of normal range
      -inf - negative infinity, parameter not defined

Re: 5520 switch, 1gbps optics in VIM, connected but no IP connectivity

Stefan_K_ — Sat, 06 Mar 2021 00:58:46 GMT

Whats the exact SFP type / part number?

But looks like a case for GTAC. Could be another bug in the 31.x firmware. This is why I will stick to the 30.7 branch as long as possible and hopefully won’t get in contact with 5520s too often...

Re: 5520 switch, 1gbps optics in VIM, connected but no IP connectivity

Keith9 — Thu, 11 Mar 2021 02:32:37 GMT

Extreme Networks I-MGBIC-GSX Compatible 1000BASE-SX SFP 850nm 550m DOM LC MMF Transceiver Module

https://www.fs.com/products/32864.html

These work perfectly in X690’s and X450G2’s, as well as other fiber optic class modules of various 1gbps wavelengths and part numbers like 10065, 10051H, etc. Also 1000BASE-T such as https://www.fs.com/products/32882.html work perfectly in every Extreme switch model I’ve tried except the 5520’s VIM-4X slots.

The 5520 series switches don’t seem to pass ANY kind of IP traffic for regular SFP’s in the VIM-4X slots. SFP+ like 10gig fiber at various wavelengths or TWINAX cables to servers running at 10gig all work fine.

Its not a port speed issue either. I tried the whole auto off speed 1000 thing too.

They are running 31.2.1.1 which fixed the autonegotiation issue for 100mbps and slower devices on the front facing 10/100/1000 baseT RJ45 ports.

They came with a variant of 31.1.1.1 which was an engineering build and had the prefix (software update required) on every CLI prompt. We upgraded to 31.1.1.2 to remove that prompt and then 31.2.1.1 to discover our lower speed devices worked then.

You might like the 30.7 branch, but the 5520’s came from the factory with a higher version and being they are in my opinon, bleeding edge, not fully baked in yet I don’t know if you can go back. Perhaps EXOS 31 was the first release to support 5520’s? They had to come with that version and not a more stable prior version for a reason, right?

Re: 5520 switch, 1gbps optics in VIM, connected but no IP connectivity

Stefan_K_ — Thu, 11 Mar 2021 02:56:44 GMT

Yes, 5520 are only supported by 31.x. That’s why I mentioned that I hope that I won’t get in contact with 5520 in the near future. 😄

According to https://optics.extremenetworks.com/ the I-MGBIC-GSX is officially compatible only with X440G2 and X435. (even though it works in other switches...)

What I didn’t understand: Have you tested an official supported SFP like 10051H in an 5520? If not, could you maybe test this (even if this doesn’t help in your case). I’m curious…

Re: 5520 switch, 1gbps optics in VIM, connected but no IP connectivity

Keith9 — Thu, 11 Mar 2021 02:59:52 GMT

Yes, 5520 are only supported by 31.x. That’s why I mentioned that I hope that I won’t get in contact with 5520 in the near future. 😄

According to https://optics.extremenetworks.com/ the I-MGBIC-GSX is officially compatible only with X440G2 and X435. (even though it works in other switches...)

What I didn’t understand: Have you tested an official supported SFP like 10051H in an 5520? If not, could you maybe test this (even if this doesn’t help in your case). I’m curious…

I could get the FS.com Extreme coded version of the 10051H. Otherwise overpriced SFP modules are not in the budget.

We can have a shoebox full of spares for the price of one “official” branded module. Not worth it IMO.

I actually have an order to place today for some fiber optic cabling for another project. I’ll throw one on, its only $7.