MPLS Kernel error 16.1.3.6


VPLS VPN not working. After disabling see error

MPLS gport resolve failed for unit = 0 vpn = 0x70e3, port = 0xffffffff vp = 0x18000a0c, rv = -7 (Entry not found)08/30/2016 11:37:25.99
extreme_mpls_del_vp_nh_xref: MPLS ECMP NH vp : 0xa0c l3intf = 0x83f map entry not found in btree rv = -7 (Entry not found)
extreme_custom_mpls_l2vpn_port_del: MPLS port delete: vp: 0xa0c l3Intf = 0x83f map entry not found in btree rv = -7 (Entry not found)
MPLS bcm_mpls_port_del failed for unit = 0 vpn = 0x70e3, vp = 0x18000a0c rv = -7 (Entry not found)


after enabling
* MPLS: bcm_mpls_port_add failed for unit = 0 vpn = 0x70e3, port = 0x8000809 vp = 0x18000a0d, rv = -14 (No resources for operation)

sh ver
Switch : Rev 12.0 BootROM: 2.0.2.1 IMG: 16.1.3.6

Image : ExtremeXOS version 16.1.3.6 16.1.3.6-patch1-9 by release-manager
on Mon Jul 18 10:45:49 EDT 2016
BootROM : 2.0.2.1
Diagnostics : 6.4

sh switch

System Type: X670-48x

Same problem resolved on previous release ?
https://gtacknowledge.extremenetworks.com/articles/Solution/Kern-MPLS-Error-No-resources-for-operati...

40 replies

Userlevel 4
MPLS kernel message you have faced logged due to unavailability of hardware resources it is similar to the another article that you have mentioned, but in that case issue appears after changing dot1q tag (include/exclude), so it is not the same issue for sure.
Also it is not clear what interface/protocol you have disabled/enabled.

In a case when your services are affected after the issue try to removed/re-add service VLAN to VPLS as a workaround for restoring connectivity.
Can you explain where in message info about vlan problem?
Now we planned downgrade to 16.1.3.6-1-8.
We already try to delete and create same vpls but problem is not disappear.
Userlevel 6
Sergey,

Did the VPN problem start after a network failure or OSPF neighborship loss?

It is probably best to open a case with GTAC so that this can be analyzed.

How to contact Extreme Networks Global Technical Assistance Center (GTAC)
Yes. Ospf flap was before first mpls error in log. Down due to BFD path down.

8/30/2016 03:44:37.98 MPLS: bcm_mpls_port_add failed for unit = 0 vpn = 0x7055, port = 0x800080a vp = 0x180005c7, rv = -14 (No resources for operation)
08/30/2016 03:44:37.98
08/30/2016 03:44:37.98 MPLS: bcm_mpls_port_add failed for unit = 0 vpn = 0x7054, port = 0x800080a vp = 0x180005c6, rv = -14 (No resources for operation)
08/30/2016 03:44:37.98
08/30/2016 03:44:37.98 MPLS: bcm_mpls_port_add failed for unit = 0 vpn = 0x7053, port = 0x800080a vp = 0x180005c5, rv = -14 (No resources for operation)
08/30/2016 03:44:37.98
08/30/2016 03:44:37.98 MPLS: bcm_mpls_port_add failed for unit = 0 vpn = 0x7052, port = 0x800080a vp = 0x180005c4, rv = -14 (No resources for operation)
08/30/2016 03:44:37.84 Changing the state of neighbor rtid 10.16.0.1 ipa 10.16.248.10 to state = FULL due to Loading done.
08/30/2016 03:44:37.84 Changing the state of neighbor rtid 10.16.0.1 ipa 10.16.248.10 to state = LOADING due to exchange done event.
08/30/2016 03:44:37.83 Changing the state of neighbor rtid 10.16.0.1 ipa 10.16.248.10 to state = EXCHANGE due to negotiation done event.
08/30/2016 03:44:36.86 Changing the state of neighbor rtid 10.16.0.1 ipa 10.16.248.10 to state = EX_START due to AdjOK event.
08/30/2016 03:44:35.86 Changing the state of neighbor rtid 10.16.0.1 ipa 10.16.248.10 to state = 2WAY due to two way event.
08/30/2016 03:44:35.86 Changing the state of neighbor rtid 10.16.0.1 ipa 10.16.248.10 to state = INIT due to hello received.
08/30/2016 03:44:35.86 Changing the state of neighbor rtid 10.16.0.1 ipa 0.0.0.0 to state = DOWN due to new neighbor.
08/30/2016 03:44:35.59 Changing the state of neighbor rtid 10.16.0.1 ipa 10.16.248.10 to state = DOWN due to BFD path down.
08/30/2016 03:32:52.66 Changing the state of neighbor rtid 10.16.16.1 ipa 10.16.248.5 to state = FULL due to Loading done.
08/30/2016 03:32:52.66 Changing the state of neighbor rtid 10.16.16.1 ipa 10.16.248.5 to state = LOADING due to exchange done event.
Userlevel 6
Sergey Vekli wrote:

Yes. Ospf flap was before first mpls error in log. Down due to BFD path down.

8/30/2016 03:44:37.98 MPLS: bcm_mpls_port_add failed for unit = 0 vpn = 0x7055, port = 0x800080a vp = 0x180005c7, rv = -14 (No resources for operation)
08/30/2016 03:44:37.98
08/30/2016 03:44:37.98 MPLS: bcm_mpls_port_add failed for unit = 0 vpn = 0x7054, port = 0x800080a vp = 0x180005c6, rv = -14 (No resources for operation)
08/30/2016 03:44:37.98
08/30/2016 03:44:37.98 MPLS: bcm_mpls_port_add failed for unit = 0 vpn = 0x7053, port = 0x800080a vp = 0x180005c5, rv = -14 (No resources for operation)
08/30/2016 03:44:37.98
08/30/2016 03:44:37.98 MPLS: bcm_mpls_port_add failed for unit = 0 vpn = 0x7052, port = 0x800080a vp = 0x180005c4, rv = -14 (No resources for operation)
08/30/2016 03:44:37.84 Changing the state of neighbor rtid 10.16.0.1 ipa 10.16.248.10 to state = FULL due to Loading done.
08/30/2016 03:44:37.84 Changing the state of neighbor rtid 10.16.0.1 ipa 10.16.248.10 to state = LOADING due to exchange done event.
08/30/2016 03:44:37.83 Changing the state of neighbor rtid 10.16.0.1 ipa 10.16.248.10 to state = EXCHANGE due to negotiation done event.
08/30/2016 03:44:36.86 Changing the state of neighbor rtid 10.16.0.1 ipa 10.16.248.10 to state = EX_START due to AdjOK event.
08/30/2016 03:44:35.86 Changing the state of neighbor rtid 10.16.0.1 ipa 10.16.248.10 to state = 2WAY due to two way event.
08/30/2016 03:44:35.86 Changing the state of neighbor rtid 10.16.0.1 ipa 10.16.248.10 to state = INIT due to hello received.
08/30/2016 03:44:35.86 Changing the state of neighbor rtid 10.16.0.1 ipa 0.0.0.0 to state = DOWN due to new neighbor.
08/30/2016 03:44:35.59 Changing the state of neighbor rtid 10.16.0.1 ipa 10.16.248.10 to state = DOWN due to BFD path down.
08/30/2016 03:32:52.66 Changing the state of neighbor rtid 10.16.16.1 ipa 10.16.248.5 to state = FULL due to Loading done.
08/30/2016 03:32:52.66 Changing the state of neighbor rtid 10.16.16.1 ipa 10.16.248.5 to state = LOADING due to exchange done event.

Sergey,

There is a known issue that this could be related to. However, this could also be a new issue. I think it is best to open up a GTAC case for investigation.
Can high CPU load cause BFD timeout?
Userlevel 6
Sergey Vekli wrote:

Can high CPU load cause BFD timeout?

Yes under certain conditions a high CPU load could cause a BFD timeout.
Case opened.
Userlevel 2
Sergey,

Is there any RSVP enabled on that path?
I suggest you to disable it in case of lose of connectivity.
We faced the same problem and we're waiting for a patch to solve this problem.
Hi. No. We don't use RSVP.
Userlevel 2
Sergey Vekli wrote:

Hi. No. We don't use RSVP.

odd.
How did you get the traffic get back to work?
My workaround for that is disabling rsvp and if it still doesn't solve that, the only way is to delete the L2VPN config and then configure again.
Sergey Vekli wrote:

Hi. No. We don't use RSVP.

We try delete and recreate and disable and enable L2VPN -no result. Now we downgrade to 1-8 and problem not appear again.
Userlevel 6
Sergey Vekli wrote:

Hi. No. We don't use RSVP.

Hi Sergey, it's difficult to say that the downgrade to patch 1-8 has fixed the issue itself. Since you had to reboot, that action cleared and then synced the switch MPLS tables (HW and SW).

If you didn't try rebooting the switch before the upgrade, a reboot itself could be fixed this issue.
Sergey Vekli wrote:

Hi. No. We don't use RSVP.

Reboot can't be solution because it have very big out of service time.
Userlevel 6
Sergey Vekli wrote:

Hi. No. We don't use RSVP.

Hi Sergey, what I meant is that sometimes an upgrade or a downgrade seems to fix the issue. However, since the upgrade process requires a reboot, just the reboot could be enough.

How is your MPLS network so far? Working as expected after the firmware downgrade?
Sergey Vekli wrote:

Hi. No. We don't use RSVP.

Yes. Problem is not apper.
Sergey Vekli wrote:

Hi. No. We don't use RSVP.

Hello guys,

Dear Sergey Vekli, the problem has been fixed in version 16.1.3.6 EXOS-patch1-8 ?

we have reported the same problem.

Firmware version 15.5.2.9
Sergey Vekli wrote:

Hi. No. We don't use RSVP.

In 1-8 version problem is not repeated.
Sergey Vekli wrote:

Hi. No. We don't use RSVP.

Now we use 1-11, for now problem is not seen.
Userlevel 6
Sergey Vekli wrote:

Hi. No. We don't use RSVP.

Hi Jader, currently the EXOS 16.1.3.6 is the recommended release for many platforms.

If you are using MPLS, there are couple improvements on EXOS 16.1.3/4 last patches.

Regarding the recommended releases you can confirm based on your platform in the article below:

https://gtacknowledge.extremenetworks.com/articles/Q_A/What-Is-The-Recommended-Release-of-EXOS-For-M...
Sergey Vekli wrote:

Hi. No. We don't use RSVP.

In EXOS 16.1.3.6 patch 9 we see another problem with VPLS. If service vman in vpls has configured on port as cep with range cvids it is not work. On 11 it work fine.
Now we again have error in log. Why BFD path down at same time for all 2 neighbors?

11/17/2016 12:22:58.12 Changing the state of neighbor rtid 10.15.0.1 ipa 10.16.248.18 to state = FULL due to Loading done.
11/17/2016 12:22:58.12 Changing the state of neighbor rtid 10.15.0.1 ipa 10.16.248.18 to state = LOADING due to exchange done event.
11/17/2016 12:22:58.12 Changing the state of neighbor rtid 10.15.0.1 ipa 10.16.248.18 to state = EXCHANGE due to negotiation done event.
11/17/2016 12:22:57.94 Changing the state of neighbor rtid 10.16.0.1 ipa 10.16.248.10 to state = FULL due to Loading done.
11/17/2016 12:22:57.94 Changing the state of neighbor rtid 10.16.0.1 ipa 10.16.248.10 to state = LOADING due to exchange done event.
11/17/2016 12:22:57.94 Changing the state of neighbor rtid 10.16.0.1 ipa 10.16.248.10 to state = EXCHANGE due to negotiation done event.
11/17/2016 12:22:57.93 Changing the state of neighbor rtid 10.16.0.1 ipa 10.16.248.10 to state = EX_START due to two way event.
11/17/2016 12:22:57.93 Changing the state of neighbor rtid 10.16.0.1 ipa 10.16.248.10 to state = INIT due to hello received.
11/17/2016 12:22:57.93 Changing the state of neighbor rtid 10.16.0.1 ipa 0.0.0.0 to state = DOWN due to new neighbor.
11/17/2016 12:22:57.78 Changing the state of neighbor rtid 10.15.0.1 ipa 10.16.248.18 to state = EX_START due to two way event.
11/17/2016 12:22:57.78 Changing the state of neighbor rtid 10.15.0.1 ipa 10.16.248.18 to state = INIT due to hello received.
11/17/2016 12:22:57.78 Changing the state of neighbor rtid 10.15.0.1 ipa 0.0.0.0 to state = DOWN due to new neighbor.
11/17/2016 12:22:57.57 Changing the state of neighbor rtid 10.15.0.1 ipa 10.16.248.18 to state = DOWN due to BFD path down.
11/17/2016 12:22:57.56 Changing the state of neighbor rtid 10.16.0.1 ipa 10.16.248.10 to state = DOWN due to BFD path down.
Userlevel 6
Did you start to see this issue after any change like configuration, firmware upgrade or that has started without any change?

Could you please share the outputs below:

show configuration | include bfd
show bfd
show bfd counters
show bfd session detail
Maybe it caused by enabling bgp for multicast. No other global changes in config.

enable bfd vlan "IP_InterEx"
configure bfd vlan "IP_InterEx" receive-interval 100 transmit-interval 100
enable bfd vlan "IP_41"
configure bfd vlan "IP_41" receive-interval 100 transmit-interval 100
configure ospf vlan IP_41 bfd on
configure ospf vlan IP_InterEx bfd on

Number of sessions : 2 Sessions in Init State : 0
Sessions in Down State : 0
Sessions in Admin Down State : 0
Sessions in Up State : 2

SNMP Traps for session-down : Disabled
SNMP Traps for session-up : Disabled
SNMP Traps for Batch Delay : 1000 ms

Valid Tx Pkt : 84045257 Valid Rx Pkt : 78958239Rx Invalid TTL : 0 Rx Invalid UDP SrcPort : 0
Interface Not found : 0 Rx Invalid Version : 0
Rx Invalid Length Pkt : 0 Rx Invalid Multiplier : 0
Rx Invalid Demand Mode : 0 Rx Poll & Final set : 0
Rx Invalid My Discriminator : 0 Rx Invalid Your Discriminator : 0
Rx Invalid Auth Length : 0 Rx session Not Found : 21
Auth Type Fails : 0 Authentication Fails : 0
Tx Fails : 0 Rx Discarded Pkt : 0

Neighbor : 10.16.248.18 Local : 10.16.248.17VR-Name : VR-Default Interface : IP_InterEx
Session Type : Single Hop State : Up
Detect Time : 300 ms Age : 50 ms
Discriminator (local/remote) : 1 / 1
Demand Mode (local/remote) : Off / Off
Poll (local/remote) : Off / Off
Tx Interval (local/remote) : 100 / 100 ms
Rx Interval (local/remote) : 100 / 100 ms
oper Tx Interval : 100 ms
oper Rx Interval : 100 ms
Multiplier (local/remote) : 3 / 3
Local Diag : 0 (No Diagnostic)
Remote Diag : 0 (No Diagnostic)
Authentication : None
Clients : OSPF
Uptime : 00 days 04 hours 00 minutes 13 seconds
Up Count : 1
Last Valid Packet Rx : 16:23:11.730392
Last Packet Tx : 16:23:11.720158

Neighbor : 10.16.248.10 Local : 10.16.248.9
VR-Name : VR-Default Interface : IP_41
Session Type : Single Hop State : Up
Detect Time : 300 ms Age : 30 ms
Discriminator (local/remote) : 2 / 15
Demand Mode (local/remote) : Off / Off
Poll (local/remote) : Off / Off
Tx Interval (local/remote) : 100 / 100 ms
Rx Interval (local/remote) : 100 / 100 ms
oper Tx Interval : 100 ms
oper Rx Interval : 100 ms
Multiplier (local/remote) : 3 / 3
Local Diag : 0 (No Diagnostic)
Remote Diag : 0 (No Diagnostic)
Authentication : None
Clients : OSPF
Uptime : 00 days 04 hours 00 minutes 13 seconds
Up Count : 4
Last Valid Packet Rx : 16:23:11.756480
Last Packet Tx : 16:23:11.730150

Reply