Header Only - DO NOT REMOVE - Extreme Networks

MPLS Kernel error 16.1.3.6


VPLS VPN not working. After disabling see error

MPLS gport resolve failed for unit = 0 vpn = 0x70e3, port = 0xffffffff vp = 0x18000a0c, rv = -7 (Entry not found)08/30/2016 11:37:25.99
extreme_mpls_del_vp_nh_xref: MPLS ECMP NH vp : 0xa0c l3intf = 0x83f map entry not found in btree rv = -7 (Entry not found)
extreme_custom_mpls_l2vpn_port_del: MPLS port delete: vp: 0xa0c l3Intf = 0x83f map entry not found in btree rv = -7 (Entry not found)
MPLS bcm_mpls_port_del failed for unit = 0 vpn = 0x70e3, vp = 0x18000a0c rv = -7 (Entry not found)


after enabling
* MPLS: bcm_mpls_port_add failed for unit = 0 vpn = 0x70e3, port = 0x8000809 vp = 0x18000a0d, rv = -14 (No resources for operation)

sh ver
Switch : Rev 12.0 BootROM: 2.0.2.1 IMG: 16.1.3.6

Image : ExtremeXOS version 16.1.3.6 16.1.3.6-patch1-9 by release-manager
on Mon Jul 18 10:45:49 EDT 2016
BootROM : 2.0.2.1
Diagnostics : 6.4

sh switch

System Type: X670-48x

Same problem resolved on previous release ?
https://gtacknowledge.extremenetworks.com/articles/Solution/Kern-MPLS-Error-No-resources-for-operati...

40 replies

What does mean "BFD path down"?After disabling BGP problem is disapear.
Userlevel 6
What does mean "BFD path down"?It's just a suggestion to check if this flapping stops...
What does mean "BFD path down"?No. Why we need to increase?
Userlevel 6
What does mean "BFD path down"?Did you increase the BFD timer?
What does mean "BFD path down"?Ok. We enable log for bfd events.
Userlevel 6
What does mean "BFD path down"?You can also add the BFD state change event in the log to track the flap as follows:

configure log filter DefaultFilter add events BFD.SessnStateChg
Userlevel 6
What does mean "BFD path down"?I'm suspecting something with CPU and lower BFD timers since both OSPF neighbors configured with BFD have converged at same time as follows:

11/17/2016 12:22:57.57 Changing the state of neighbor rtid 10.15.0.1 ipa 10.16.248.18 to state = DOWN due to BFD path down.

11/17/2016 12:22:57.56 Changing the state of neighbor rtid 10.16.0.1 ipa 10.16.248.10 to state = DOWN due to BFD path down.
Userlevel 6
What does mean "BFD path down"?Since the BFD is handled by the switch CPU, when using lower timers, if you have any CPU spike it could affect BFD.

If the issue occurs 3-4 times/day then you can try to adjust/increase the timer and check if the behavior changes.
What does mean "BFD path down"? debug bfd show session

Session ID: 1 Remote ID: 1 srcPort: 49152 TxSock: 11 client count: 1
destIP: 10.16.248.18, srcIP: 10.16.248.17, vlan: IP_InterEx
sessionToBeDeleted: FALSE sessnDelAborted: FALSE numPduRemain: 0 delReqPeerId: -1 delReqClientId: -1
countDelInits: 0 countDelAborts: 0
Event RECIVED_ADMIN_DOWN = 0
Event RECEIVED_DOWN = 0
Event RECEIVED_INIT = 1
Event RECEIVED_UP = 892908
Event DETECT_TIMEOUT = 0
Event ECHO_FAILED = 0
Event LOCAL_ADMINDOWN = 0
Event LOCAL_ADMINUP = 0
Event LOCAL_TIMER_CHANGE = 0
Client ID : 8193,

Session ID: 2 Remote ID: 15 srcPort: 49153 TxSock: 11 client count: 1
destIP: 10.16.248.10, srcIP: 10.16.248.9, vlan: IP_41
sessionToBeDeleted: FALSE sessnDelAborted: TRUE numPduRemain: 0 delReqPeerId: -1 delReqClientId: -1
countDelInits: 3 countDelAborts: 3
Event RECIVED_ADMIN_DOWN = 0
Event RECEIVED_DOWN = 8
Event RECEIVED_INIT = 3
Event RECEIVED_UP = 37585947
Event DETECT_TIMEOUT = 3
Event ECHO_FAILED = 0
Event LOCAL_ADMINDOWN = 3
Event LOCAL_ADMINUP = 3
Event LOCAL_TIMER_CHANGE = 0
Client ID : 8193,
What does mean "BFD path down"?As i can see timeout is not reached or i am wrong? Why you think about increase the timers?
Userlevel 6
What does mean "BFD path down"?You can add some BFD log events to try to get additional information.

Please take a look at the Article below:

https://gtacknowledge.extremenetworks.com/articles/How_To/How-to-enable-additional-debug-logs-in-EXO...

I would recommend you to increase the timer a little from 100ms to 150ms/200ms and check if this flap stops.
What does mean "BFD path down"?Problem repeated 2-3 times per day. What command can debug bfd problem?
What does mean "BFD path down"?Yes. Both device connected directly. One of them X670 other Cisco Me3600X.
How it can be what two difference device directly conected in different ports can't send bfd packet at the same time? All devices connected in ring but problem seen only on one I don't think it is a network problem.
Userlevel 6
What does mean "BFD path down"?For some reason, the BFD did not receive the health check packet on time (that could be due to many reasons.). When that happened the BFD state went down and then the OSPF has converged.

Based on the outputs provided the BFD is configured to 300ms and applied to both OSPF vlans "IP_41" and "IP_InterEx".

Are those vlans/ports direct connected to the OSPF neighbor switch/router?
What does mean "BFD path down"?
Maybe it caused by enabling bgp for multicast. No other global changes in config.

enable bfd vlan "IP_InterEx"
configure bfd vlan "IP_InterEx" receive-interval 100 transmit-interval 100
enable bfd vlan "IP_41"
configure bfd vlan "IP_41" receive-interval 100 transmit-interval 100
configure ospf vlan IP_41 bfd on
configure ospf vlan IP_InterEx bfd on

Number of sessions : 2 Sessions in Init State : 0
Sessions in Down State : 0
Sessions in Admin Down State : 0
Sessions in Up State : 2

SNMP Traps for session-down : Disabled
SNMP Traps for session-up : Disabled
SNMP Traps for Batch Delay : 1000 ms

Valid Tx Pkt : 84045257 Valid Rx Pkt : 78958239Rx Invalid TTL : 0 Rx Invalid UDP SrcPort : 0
Interface Not found : 0 Rx Invalid Version : 0
Rx Invalid Length Pkt : 0 Rx Invalid Multiplier : 0
Rx Invalid Demand Mode : 0 Rx Poll & Final set : 0
Rx Invalid My Discriminator : 0 Rx Invalid Your Discriminator : 0
Rx Invalid Auth Length : 0 Rx session Not Found : 21
Auth Type Fails : 0 Authentication Fails : 0
Tx Fails : 0 Rx Discarded Pkt : 0

Neighbor : 10.16.248.18 Local : 10.16.248.17VR-Name : VR-Default Interface : IP_InterEx
Session Type : Single Hop State : Up
Detect Time : 300 ms Age : 50 ms
Discriminator (local/remote) : 1 / 1
Demand Mode (local/remote) : Off / Off
Poll (local/remote) : Off / Off
Tx Interval (local/remote) : 100 / 100 ms
Rx Interval (local/remote) : 100 / 100 ms
oper Tx Interval : 100 ms
oper Rx Interval : 100 ms
Multiplier (local/remote) : 3 / 3
Local Diag : 0 (No Diagnostic)
Remote Diag : 0 (No Diagnostic)
Authentication : None
Clients : OSPF
Uptime : 00 days 04 hours 00 minutes 13 seconds
Up Count : 1
Last Valid Packet Rx : 16:23:11.730392
Last Packet Tx : 16:23:11.720158

Neighbor : 10.16.248.10 Local : 10.16.248.9
VR-Name : VR-Default Interface : IP_41
Session Type : Single Hop State : Up
Detect Time : 300 ms Age : 30 ms
Discriminator (local/remote) : 2 / 15
Demand Mode (local/remote) : Off / Off
Poll (local/remote) : Off / Off
Tx Interval (local/remote) : 100 / 100 ms
Rx Interval (local/remote) : 100 / 100 ms
oper Tx Interval : 100 ms
oper Rx Interval : 100 ms
Multiplier (local/remote) : 3 / 3
Local Diag : 0 (No Diagnostic)
Remote Diag : 0 (No Diagnostic)
Authentication : None
Clients : OSPF
Uptime : 00 days 04 hours 00 minutes 13 seconds
Up Count : 4
Last Valid Packet Rx : 16:23:11.756480
Last Packet Tx : 16:23:11.730150
Userlevel 6
Did you start to see this issue after any change like configuration, firmware upgrade or that has started without any change?

Could you please share the outputs below:

show configuration | include bfd
show bfd
show bfd counters
show bfd session detail
Now we again have error in log. Why BFD path down at same time for all 2 neighbors?

11/17/2016 12:22:58.12 Changing the state of neighbor rtid 10.15.0.1 ipa 10.16.248.18 to state = FULL due to Loading done.
11/17/2016 12:22:58.12 Changing the state of neighbor rtid 10.15.0.1 ipa 10.16.248.18 to state = LOADING due to exchange done event.
11/17/2016 12:22:58.12 Changing the state of neighbor rtid 10.15.0.1 ipa 10.16.248.18 to state = EXCHANGE due to negotiation done event.
11/17/2016 12:22:57.94 Changing the state of neighbor rtid 10.16.0.1 ipa 10.16.248.10 to state = FULL due to Loading done.
11/17/2016 12:22:57.94 Changing the state of neighbor rtid 10.16.0.1 ipa 10.16.248.10 to state = LOADING due to exchange done event.
11/17/2016 12:22:57.94 Changing the state of neighbor rtid 10.16.0.1 ipa 10.16.248.10 to state = EXCHANGE due to negotiation done event.
11/17/2016 12:22:57.93 Changing the state of neighbor rtid 10.16.0.1 ipa 10.16.248.10 to state = EX_START due to two way event.
11/17/2016 12:22:57.93 Changing the state of neighbor rtid 10.16.0.1 ipa 10.16.248.10 to state = INIT due to hello received.
11/17/2016 12:22:57.93 Changing the state of neighbor rtid 10.16.0.1 ipa 0.0.0.0 to state = DOWN due to new neighbor.
11/17/2016 12:22:57.78 Changing the state of neighbor rtid 10.15.0.1 ipa 10.16.248.18 to state = EX_START due to two way event.
11/17/2016 12:22:57.78 Changing the state of neighbor rtid 10.15.0.1 ipa 10.16.248.18 to state = INIT due to hello received.
11/17/2016 12:22:57.78 Changing the state of neighbor rtid 10.15.0.1 ipa 0.0.0.0 to state = DOWN due to new neighbor.
11/17/2016 12:22:57.57 Changing the state of neighbor rtid 10.15.0.1 ipa 10.16.248.18 to state = DOWN due to BFD path down.
11/17/2016 12:22:57.56 Changing the state of neighbor rtid 10.16.0.1 ipa 10.16.248.10 to state = DOWN due to BFD path down.
Hi. No. We don't use RSVP.

In EXOS 16.1.3.6 patch 9 we see another problem with VPLS. If service vman in vpls has configured on port as cep with range cvids it is not work. On 11 it work fine.
Userlevel 6
Hi. No. We don't use RSVP.

Hi Jader, currently the EXOS 16.1.3.6 is the recommended release for many platforms.

If you are using MPLS, there are couple improvements on EXOS 16.1.3/4 last patches.

Regarding the recommended releases you can confirm based on your platform in the article below:

https://gtacknowledge.extremenetworks.com/articles/Q_A/What-Is-The-Recommended-Release-of-EXOS-For-M...
Hi. No. We don't use RSVP.

Now we use 1-11, for now problem is not seen.
Hi. No. We don't use RSVP.

In 1-8 version problem is not repeated.
Hi. No. We don't use RSVP.

Hello guys,

Dear Sergey Vekli, the problem has been fixed in version 16.1.3.6 EXOS-patch1-8 ?

we have reported the same problem.

Firmware version 15.5.2.9
Hi. No. We don't use RSVP.

Yes. Problem is not apper.
Userlevel 6
Hi. No. We don't use RSVP.

Hi Sergey, what I meant is that sometimes an upgrade or a downgrade seems to fix the issue. However, since the upgrade process requires a reboot, just the reboot could be enough.

How is your MPLS network so far? Working as expected after the firmware downgrade?

Reply