5520 switch, 1gbps optics in VIM, connected but no IP connectivity

[Keith9]

I have a new 2 switch stack of 5520’s each with VIM-4X in the right side.  I have 2 10gbps fibers that work perfectly in a load sharing to connect to another floor, and 2 Active Twinax to a vmware server in a load sharing link working perfectly.

I have a wireless rooftop backup antenna, its only backup so not mission critical as I have a 1gbps WAN and a 100mbps WAN already in this building, but I noticed no IP connectivity to this over the 1gbps fiber.

Port 1:58 shows 1000 FULL connection.  The port link lights are up.
The port is in untagged vlan P2P-WIRELESS, and has an ip of 192.168.4.2/28.

The radio on the roof has an ip of 192.168.4.5/28 and is NOT pingable.

Neither is the other side of the link 192.168.4.1/28 at our HQ building about 500 feet away.  No OSPF adjacency has formed becuase even though we have Layer 1 link, it doesn’t seem like traffic likes to pass on 1gbps SFP’s in VIM4X’s on 5520’s running EXOS 31.2.1.1.

sh iparp port 1:58 shows no mac entries,  sh nodealias ports 1:58 shows nothing either.

I did disable port 1:58 then enable port 1:58, in the log it shows this…

03/05/2021 10:30:50.85 <Info:vlan.msgs.portLinkStateUp> Port 1:58 link UP at speed 1 Gbps and full-duplex

So link up but IP traffic won’t traverse.

sh port 1:58 vid
         Untagged
Port     /Tagged   VID(s)
-------- --------  ------------------------------------------------------------
1:58     Untagged  44
vlan info

P2P-WIRELESS    44   192.168.4.2    /28  -f-------o----------------    ANY    1 /1   VR-Default

VLAN Interface with name P2P-WIRELESS created by user
    Admin State:         Enabled     Tagging:   802.1Q Tag 44
    Description:         None
    Virtual router:      VR-Default
    IP Anycast:          Disabled
    IPv4 Forwarding:     Enabled
    IPv4 MC Forwarding:  Disabled
    Primary IP:          192.168.4.2/28
    IPv6 Forwarding:     Disabled
    IPv6 MC Forwarding:  Disabled
    IPv6:                None
    STPD:                None
    Protocol:            Match all unfiltered protocols
    Loopback:            Disabled
    NetLogin:            Disabled
    QosProfile:          None configured
    Egress Rate Limit Designated Port: None configured
    Flood Rate Limit QosProfile:       None configured
    Suppress ARP:        Disabled
    Suppress ND:         Disabled
    Proxy ARP:           Entry required
    Ports:   1.           (Number of active ports=1)
       Untag:   *1:58
        Flags:    (*) Active, (!) Disabled, (g) Load Sharing port
                  (b) Port blocked on the vlan, (m) Mac-Based port
                  (i) Port inactivated on the vlan due to VXLAN configuration
                  (a) Egress traffic allowed for NetLogin
                  (u) Egress traffic unallowed for NetLogin
                  (t) Translate VLAN tag for Private-VLAN
                  (s) Private-VLAN System Port, (L) Loopback port
                  (x) VMAN Tag Translated port
                  (A) Dynamically added by Auto-peering
                  (F) Dynamically added by Fabric Attach
                  (G) Multi-switch LAG Group port
                  (H) Dynamically added by MVRP
                  (I) Dynamically added by IDM
                  (N) Dynamically added by Netlogin
                  (U) Dynamically added uplink port
                  (V) Dynamically added by VM Tracking
pinging the device on the roof (which obvisouly has power since the laser is working in the sfp...) times out.  Here’s the fiber signal levels for this 1gbps sfp:

sh port 1:58 transceiver information

Port      Temp    TxPower  RxPower  TxBiasCurrent  Voltage-Aux1/ Voltage-Aux2
       (Celsius)   (dBm)    (dBm)     (mA)         Vcc (Volts)       (Volts)
================================================================================
1:58      27.83    -4.56    -14.24    2.63          3.37          N/A
================================================================================
      N/A  - parameter not applicable to transceiver connected to the port
      D    - Optic supports Digital Diagnostic Monitoring Interface (DDMI) which is off
      *    - value is out of normal range
      -inf - negative infinity, parameter not defined
 

So how can one have layer 1 link but no IP connectivity?

The Cisco 3750X switch stack that was here previously had no problem with a Cisco SFP in sled Gi1/1/1.

Here is the previous Cisco config

interface GigabitEthernet1/1/1
 description LightPointe FSO Link
 switchport access vlan 44
 ip arp inspection trust
 ip access-group acl1 in
 mls qos trust dscp
 spanning-tree portfast edge
 ip dhcp snooping trust

Here’s everything in EXOS having to do with port 1:58
sh configuration | i 1:58
configure vlan default delete ports 1:2,1:45,1:47,1:58,2:48,2:57,2:59
enable diffserv examination port 1:58
disable dot1p examination port 1:58
configure ports 1:58 description-string "P2P Wireless connection to HQ VL44"
configure ports 1:58 auto off speed 1000 duplex full
configure vlan P2P-WIRELESS add ports 1:58 untagged
configure qosprofile QP3 minbw 0 maxbw 100 ports 1:58
configure qosprofile QP5 minbw 0 maxbw 100 ports 1:58
disable edp ports 1:58
enable cdp ports 1:58
enable nodealias ports 1:58

And for the vlan

sh configuration | i P2P-WIRELESS
create vlan "P2P-WIRELESS"
configure vlan P2P-WIRELESS tag 44
configure vlan P2P-WIRELESS add ports 1:58 untagged
configure vlan P2P-WIRELESS ipaddress 192.168.4.2 255.255.255.240
enable ipforwarding vlan P2P-WIRELESS
enable bootprelay ipv4 vlan P2P-WIRELESS
configure ospf add vlan P2P-WIRELESS area 0.0.0.0
configure ospf vlan P2P-WIRELESS cost 15
configure ospf vlan P2P-WIRELESS priority 0
configure ospf vlan P2P-WIRELESS authentication encrypted md5 1 "encrypted string"


This should be pretty cut and dry, shouldnt it?

[Keith9]

I ping from the switch itself.

just ping 192.168.4.5

or even

ping 192.168.4.5 from 192.168.4.2
Ping(ICMP) 192.168.4.5: 4 packets, 8 data bytes, interval 1 second(s).
Request timed out
Request timed out
Request timed out
44 bytes from 192.168.4.2: icmp_seq=0 Destination Host Unreachable
44 bytes from 192.168.4.2: icmp_seq=1 Destination Host Unreachable
44 bytes from 192.168.4.2: icmp_seq=2 Destination Host Unreachable
44 bytes from 192.168.4.2: icmp_seq=3 Destination Host Unreachable

sh iparp vlan P2P-WIRELESS
VR            Destination      Mac                Age  Static  VLAN          VID   Port

Dynamic Entries  :         129             Static Entries            :          0
Pending Entries  :           5

ARP address check:    Enabled              ARP refresh               :    Enabled
Timeout          :          20 minutes     ARP Sender-Mac Learning   :   Disabled
Locktime         :        1000 milliseconds
Retransmit Time  :        1000 milliseconds
Reachable Time   :      900000 milliseconds (Auto)
Fast Convergence :         Off

ARP Global Settings
Max Entries         :    12288
Max Pending Entries :      256
Max Proxy Entries   :      256

On the other end at HQ, a EXOS X690 core I have no problem hitting the radio head to this exterier building.

That shows me what I know prior to last nights cutover from Cisco 3750X to Extreme 5520s at the remote building.  The radio heads are working fine.

Because its goes to the roof, its fiber run.  The fiber is powered by a POE injector plugged into a PDU that also runs to the roof with it.  The fiber on the roof is into a 1000mbps 850nm mm optic.  This optic works fine in X690’s, Cisco 3750, Extreme 450-G2’s but for some reason although it links up in a 5520, traffic cannot traverse it.

We call this vlan VL44 at the HQ.

Here’s the iparp from the core at the HQ building.

-Yellow-Core.3 # sh iparp vlan VL44
VR            Destination      Mac                Age  Static  VLAN          VID   Port
VR-Default    192.168.4.3      00:60:35:1f:9c:56    3      NO  VL44          44    1:4
VR-Default    192.168.4.4      04:18:d6:58:c8:60   20      NO  VL44          44    1:4
VR-Default    192.168.4.5      00:60:35:1f:9c:65    1      NO  VL44          44    1:4
VR-Default    192.168.4.6      44:d9:e7:4a:6c:75   20      NO  VL44          44    1:4
VR-Default    192.168.4.7      02:04:96:9d:52:ac   20      NO  VL44          44    1:49

Dynamic Entries  :         450             Static Entries            :          0
Pending Entries  :          11
In Request       :  3698897465             In Response               :   23873222
Out Request      :   879593960             Out Response              :   92638294
Failed Requests  :   143335743
Proxy Answered   :      835064
Rx Error         :     1705231             Dup IP Addr               :     192.168.4.1
Rejected Count   :     8104367             Rejected IP               :  169.254.252.38
Rejected Port    :        1:49             Rejected I/F              : Default

Max ARP entries  :        8192             Max ARP pending entries   :        256
ARP address check:    Enabled              ARP refresh               :    Enabled
Timeout          :          20 minutes     ARP Sender-Mac Learning   :   Disabled
Locktime         :        1000 milliseconds
Retransmit Time  :        1000 milliseconds
Reachable Time   :      900000 milliseconds (Auto)
Fast Convergence :         Off

Its not seing 192.168.4.2 at all.

sh vlan VL44
VLAN Interface with name VL44 created by user
    Admin State:         Enabled     Tagging:   802.1Q Tag 44
    Description:         HQ to 1566 Transport
    Virtual router:      VR-Default
    IPv4 Forwarding:     Enabled
    IPv4 MC Forwarding:  Disabled
    Primary IP:          192.168.4.8/28
    IPv6 Forwarding:     Disabled
    IPv6 MC Forwarding:  Disabled
    IPv6:                None
    STPD:                s0(Enabled,Auto-bind)
    Protocol:            Match all unfiltered protocols
    Loopback:            Disabled
    NetLogin:            Disabled
    OpenFlow:            Disabled
    QosProfile:          None configured
    Egress Rate Limit Designated Port: None configured
    Flood Rate Limit QosProfile:       None configured
    Suppress ARP:        Disabled
    Proxy ARP:           Entry required
    Ports:   2.           (Number of active ports=2)
       Untag:    *1:4
       Tag:    *1:49g
        Flags:    (*) Active, (!) Disabled, (g) Load Sharing port
                  (b) Port blocked on the vlan, (m) Mac-Based port
                  (i) Port inactivated on the vlan due to VXLAN configuration
                  (a) Egress traffic allowed for NetLogin
                  (u) Egress traffic unallowed for NetLogin
                  (t) Translate VLAN tag for Private-VLAN
                  (s) Private-VLAN System Port, (L) Loopback port
                  (x) VMAN Tag Translated port
                  (A) Dynamically added by Auto-peering
                  (F) Dynamically added by Fabric Attach
                  (G) Multi-switch LAG Group port
                  (H) Dynamically added by MVRP
                  (I) Dynamically added by IDM
                  (N) Dynamically added by Netlogin
                  (U) Dynamically added uplink port
                  (V) Dynamically added by VM Tracking

Yellow-Core.7 # sh port 1:4 transceiver information

Port      Temp    TxPower  RxPower  TxBiasCurrent  Voltage-Aux1/ Voltage-Aux2
       (Celsius)   (dBm)    (dBm)     (mA)         Vcc (Volts)       (Volts)
================================================================================
1:4       28.84    -5.86    -7.05     2.64          3.36          N/A
================================================================================
      N/A  - parameter not applicable to transceiver connected to the port
      *    - value is out of normal range
      -inf - negative infinity, parameter not defined

sh

[Stefan_K_]

From where do you ping? Directly from the switch? (using the right VR?)

Do you see any arp-entries on this vlan? 

show iparp vlan P2P-WIRELESS