MPLS Kernel error 16.1.3.6

  • 0
  • 1
  • Problem
  • Updated 2 years ago
  • Solved
  • (Edited)
VPLS VPN not working. After disabling see error

<Erro:Kern.MPLS.Error> MPLS gport resolve failed for unit = 0 vpn = 0x70e3, port = 0xffffffff vp = 0x18000a0c, rv = -7 (Entry not found)08/30/2016 11:37:25.99 <Erro:Kern.Error>
<Erro:Kern.MPLS.Error> extreme_mpls_del_vp_nh_xref: MPLS ECMP NH vp : 0xa0c l3intf = 0x83f map entry not found in btree rv = -7 (Entry not found)
 <Erro:Kern.MPLS.Error> extreme_custom_mpls_l2vpn_port_del: MPLS port delete: vp: 0xa0c  l3Intf = 0x83f map entry not found in btree rv = -7 (Entry not found)
 <Erro:Kern.MPLS.Error> MPLS bcm_mpls_port_del failed for unit = 0 vpn = 0x70e3, vp = 0x18000a0c rv = -7 (Entry not found)
<Erro:Kern.Error>

after enabling
* <Erro:Kern.MPLS.Error> MPLS: bcm_mpls_port_add failed for unit = 0 vpn = 0x70e3, port = 0x8000809 vp = 0x18000a0d, rv = -14 (No resources for operation)

sh ver
Switch      :  Rev 12.0 BootROM: 2.0.2.1    IMG: 16.1.3.6


Image   : ExtremeXOS version 16.1.3.6 16.1.3.6-patch1-9 by release-manager
          on Mon Jul 18 10:45:49 EDT 2016
BootROM : 2.0.2.1
Diagnostics : 6.4

sh switch

System Type:      X670-48x



Same problem resolved on previous release ?
https://gtacknowledge.extremenetworks.com/articles/Solution/Kern-MPLS-Error-No-resources-for-operati...
Photo of Sergey Vekli

Sergey Vekli

  • 506 Points 500 badge 2x thumb

Posted 2 years ago

  • 0
  • 1
Photo of Necheporenko, Nikolay

Necheporenko, Nikolay, Employee

  • 1,480 Points 1k badge 2x thumb
MPLS kernel message you have faced logged due to unavailability of hardware resources it is similar to the another article that you have mentioned, but in that case issue appears after changing dot1q tag (include/exclude), so it is not the same issue for sure.
Also it is not clear what interface/protocol you have disabled/enabled.

In a case when your services are affected after the issue try to removed/re-add service VLAN to VPLS as a workaround for restoring connectivity.
Photo of Sergey Vekli

Sergey Vekli

  • 506 Points 500 badge 2x thumb
Can you explain where in message info about vlan problem?
Now we planned  downgrade to 16.1.3.6-1-8.
Photo of Sergey Vekli

Sergey Vekli

  • 506 Points 500 badge 2x thumb
We already try to delete and create same vpls but problem is not disappear.
Photo of Chad Smith

Chad Smith, Alum

  • 5,640 Points 5k badge 2x thumb
Sergey,

Did the VPN problem start after a network failure or OSPF neighborship loss?

It is probably best to open a case with GTAC so that this can be analyzed.

How to contact Extreme Networks Global Technical Assistance Center (GTAC)
(Edited)
Photo of Sergey Vekli

Sergey Vekli

  • 506 Points 500 badge 2x thumb
Yes. Ospf flap was before first mpls error in log. Down due to BFD path down.

8/30/2016 03:44:37.98 <Erro:Kern.MPLS.Error> MPLS: bcm_mpls_port_add failed for unit = 0 vpn = 0x7055, port = 0x800080a vp = 0x180005c7, rv = -14 (No resources for operation)
08/30/2016 03:44:37.98 <Erro:Kern.Error>
08/30/2016 03:44:37.98 <Erro:Kern.MPLS.Error> MPLS: bcm_mpls_port_add failed for unit = 0 vpn = 0x7054, port = 0x800080a vp = 0x180005c6, rv = -14 (No resources for operation)
08/30/2016 03:44:37.98 <Erro:Kern.Error>
08/30/2016 03:44:37.98 <Erro:Kern.MPLS.Error> MPLS: bcm_mpls_port_add failed for unit = 0 vpn = 0x7053, port = 0x800080a vp = 0x180005c5, rv = -14 (No resources for operation)
08/30/2016 03:44:37.98 <Erro:Kern.Error>
08/30/2016 03:44:37.98 <Erro:Kern.MPLS.Error> MPLS: bcm_mpls_port_add failed for unit = 0 vpn = 0x7052, port = 0x800080a vp = 0x180005c4, rv = -14 (No resources for operation)
 08/30/2016 03:44:37.84 <Noti:ospf.neighbor.ChgState> Changing the state of neighbor rtid 10.16.0.1 ipa 10.16.248.10 to state = FULL due to Loading done.
08/30/2016 03:44:37.84 <Noti:ospf.neighbor.ChgState> Changing the state of neighbor rtid 10.16.0.1 ipa 10.16.248.10 to state = LOADING due to exchange done event.
08/30/2016 03:44:37.83 <Noti:ospf.neighbor.ChgState> Changing the state of neighbor rtid 10.16.0.1 ipa 10.16.248.10 to state = EXCHANGE due to negotiation done event.
08/30/2016 03:44:36.86 <Noti:ospf.neighbor.ChgState> Changing the state of neighbor rtid 10.16.0.1 ipa 10.16.248.10 to state = EX_START due to AdjOK event.
08/30/2016 03:44:35.86 <Noti:ospf.neighbor.ChgState> Changing the state of neighbor rtid 10.16.0.1 ipa 10.16.248.10 to state = 2WAY due to two way event.
08/30/2016 03:44:35.86 <Noti:ospf.neighbor.ChgState> Changing the state of neighbor rtid 10.16.0.1 ipa 10.16.248.10 to state = INIT due to hello received.
08/30/2016 03:44:35.86 <Noti:ospf.neighbor.ChgState> Changing the state of neighbor rtid 10.16.0.1 ipa 0.0.0.0 to state = DOWN due to new neighbor.
08/30/2016 03:44:35.59 <Noti:ospf.neighbor.ChgState> Changing the state of neighbor rtid 10.16.0.1 ipa 10.16.248.10 to state = DOWN due to BFD path down.
08/30/2016 03:32:52.66 <Noti:ospf.neighbor.ChgState> Changing the state of neighbor rtid 10.16.16.1 ipa 10.16.248.5 to state = FULL due to Loading done.
08/30/2016 03:32:52.66 <Noti:ospf.neighbor.ChgState> Changing the state of neighbor rtid 10.16.16.1 ipa 10.16.248.5 to state = LOADING due to exchange done event.
Photo of Chad Smith

Chad Smith, Alum

  • 5,640 Points 5k badge 2x thumb
Sergey,

There is a known issue that this could be related to.  However, this could also be a new issue.  I think it is best to open up a GTAC case for investigation.
Photo of Sergey Vekli

Sergey Vekli

  • 506 Points 500 badge 2x thumb
Can high CPU load cause BFD timeout?
Photo of Chad Smith

Chad Smith, Alum

  • 5,640 Points 5k badge 2x thumb
Yes under certain conditions a high CPU load could cause a BFD timeout.
Photo of Sergey Vekli

Sergey Vekli

  • 506 Points 500 badge 2x thumb
Case opened.
Photo of Julian Eble

Julian Eble

  • 1,394 Points 1k badge 2x thumb
Sergey,

Is there any RSVP enabled on that path?
I suggest you to disable it in case of lose of connectivity.
We faced the same problem and we`re waiting for a patch to solve this problem.
Photo of Sergey Vekli

Sergey Vekli

  • 506 Points 500 badge 2x thumb
Hi. No. We don't use RSVP.
Photo of Jader Rosa

Jader Rosa

  • 60 Points
Hello guys,

Dear Sergey Vekli, the problem has been fixed in version 16.1.3.6 EXOS-patch1-8 ?


we have reported the same problem.

Firmware version 15.5.2.9
Photo of Sergey Vekli

Sergey Vekli

  • 506 Points 500 badge 2x thumb
In 1-8 version problem is not repeated.
Photo of Sergey Vekli

Sergey Vekli

  • 506 Points 500 badge 2x thumb
Now we use 1-11, for now problem is not seen.
Photo of Henrique

Henrique, Employee

  • 10,302 Points 10k badge 2x thumb
Hi Jader, currently the EXOS 16.1.3.6 is the recommended release for many platforms.

If you are using MPLS, there are couple improvements on EXOS 16.1.3/4 last patches.

Regarding the recommended releases you can confirm based on your platform in the article below:

https://gtacknowledge.extremenetworks.com/articles/Q_A/What-Is-The-Recommended-Release-of-EXOS-For-M...
Photo of Sergey Vekli

Sergey Vekli

  • 506 Points 500 badge 2x thumb
In EXOS 16.1.3.6 patch 9  we see another problem with VPLS. If service vman in vpls has configured on port as cep with range cvids it is not work. On 11 it work fine.
Photo of Sergey Vekli

Sergey Vekli

  • 506 Points 500 badge 2x thumb
Now we again have error in log. Why BFD path down at same time for all 2 neighbors? 

11/17/2016 12:22:58.12 <Noti:ospf.neighbor.ChgState> Changing the state of neighbor rtid 10.15.0.1 ipa 10.16.248.18 to state = FULL due to Loading done.
11/17/2016 12:22:58.12 <Noti:ospf.neighbor.ChgState> Changing the state of neighbor rtid 10.15.0.1 ipa 10.16.248.18 to state = LOADING due to exchange done event.
11/17/2016 12:22:58.12 <Noti:ospf.neighbor.ChgState> Changing the state of neighbor rtid 10.15.0.1 ipa 10.16.248.18 to state = EXCHANGE due to negotiation done event.
11/17/2016 12:22:57.94 <Noti:ospf.neighbor.ChgState> Changing the state of neighbor rtid 10.16.0.1 ipa 10.16.248.10 to state = FULL due to Loading done.
11/17/2016 12:22:57.94 <Noti:ospf.neighbor.ChgState> Changing the state of neighbor rtid 10.16.0.1 ipa 10.16.248.10 to state = LOADING due to exchange done event.
11/17/2016 12:22:57.94 <Noti:ospf.neighbor.ChgState> Changing the state of neighbor rtid 10.16.0.1 ipa 10.16.248.10 to state = EXCHANGE due to negotiation done event.
11/17/2016 12:22:57.93 <Noti:ospf.neighbor.ChgState> Changing the state of neighbor rtid 10.16.0.1 ipa 10.16.248.10 to state = EX_START due to two way event.
11/17/2016 12:22:57.93 <Noti:ospf.neighbor.ChgState> Changing the state of neighbor rtid 10.16.0.1 ipa 10.16.248.10 to state = INIT due to hello received.
11/17/2016 12:22:57.93 <Noti:ospf.neighbor.ChgState> Changing the state of neighbor rtid 10.16.0.1 ipa 0.0.0.0 to state = DOWN due to new neighbor.
11/17/2016 12:22:57.78 <Noti:ospf.neighbor.ChgState> Changing the state of neighbor rtid 10.15.0.1 ipa 10.16.248.18 to state = EX_START due to two way event.
11/17/2016 12:22:57.78 <Noti:ospf.neighbor.ChgState> Changing the state of neighbor rtid 10.15.0.1 ipa 10.16.248.18 to state = INIT due to hello received.
11/17/2016 12:22:57.78 <Noti:ospf.neighbor.ChgState> Changing the state of neighbor rtid 10.15.0.1 ipa 0.0.0.0 to state = DOWN due to new neighbor.
11/17/2016 12:22:57.57 <Noti:ospf.neighbor.ChgState> Changing the state of neighbor rtid 10.15.0.1 ipa 10.16.248.18 to state = DOWN due to BFD path down.
11/17/2016 12:22:57.56 <Noti:ospf.neighbor.ChgState> Changing the state of neighbor rtid 10.16.0.1 ipa 10.16.248.10 to state = DOWN due to BFD path down.
(Edited)
Photo of Henrique

Henrique, Employee

  • 10,302 Points 10k badge 2x thumb
Did you start to see this issue after any change like configuration, firmware upgrade or that has started without any change?

Could you please share the outputs below:

show configuration | include bfd
show bfd
show bfd counters
show bfd session detail
Photo of Sergey Vekli

Sergey Vekli

  • 506 Points 500 badge 2x thumb
Maybe it caused by enabling bgp for multicast. No other global changes in config.

enable bfd vlan "IP_InterEx"
configure bfd vlan "IP_InterEx" receive-interval 100 transmit-interval 100
enable bfd vlan "IP_41"
configure bfd vlan "IP_41" receive-interval 100 transmit-interval 100
configure ospf vlan IP_41 bfd on
configure ospf vlan IP_InterEx bfd on

Number of  sessions           : 2 Sessions in Init State        : 0
 Sessions in Down State        : 0
 Sessions in Admin Down State  : 0
 Sessions in Up State          : 2

 SNMP Traps for session-down   : Disabled
 SNMP Traps for session-up     : Disabled
 SNMP Traps for Batch Delay    : 1000 ms


Valid Tx Pkt                     : 84045257  Valid Rx Pkt                   : 78958239Rx Invalid TTL                   : 0         Rx Invalid UDP SrcPort         : 0
Interface Not found              : 0         Rx Invalid Version             : 0
Rx Invalid Length Pkt            : 0         Rx Invalid Multiplier          : 0
Rx Invalid Demand Mode           : 0         Rx Poll & Final set            : 0
Rx Invalid My Discriminator      : 0         Rx Invalid Your Discriminator  : 0
Rx Invalid Auth Length           : 0         Rx session Not Found           : 21
Auth Type Fails                  : 0         Authentication Fails           : 0
Tx Fails                         : 0         Rx Discarded Pkt               : 0




Neighbor       : 10.16.248.18        Local       : 10.16.248.17VR-Name        : VR-Default          Interface   : IP_InterEx
Session Type   : Single Hop          State       : Up
Detect Time    : 300 ms              Age         : 50 ms
Discriminator (local/remote)    : 1 / 1
Demand Mode (local/remote)      : Off / Off
Poll (local/remote)             : Off / Off
Tx Interval (local/remote)      : 100 / 100 ms
Rx Interval (local/remote)      : 100 / 100 ms
oper Tx Interval                : 100 ms
oper Rx Interval                : 100 ms
Multiplier (local/remote)       : 3 / 3
Local Diag                      : 0 (No Diagnostic)
Remote Diag                     : 0 (No Diagnostic)
Authentication                  : None
Clients                         : OSPF
Uptime                          : 00 days 04 hours 00 minutes 13 seconds
Up Count                        : 1
Last Valid Packet Rx            : 16:23:11.730392
Last Packet Tx                  : 16:23:11.720158

Neighbor       : 10.16.248.10        Local       : 10.16.248.9
VR-Name        : VR-Default          Interface   : IP_41
Session Type   : Single Hop          State       : Up
Detect Time    : 300 ms              Age         : 30 ms
Discriminator (local/remote)    : 2 / 15
Demand Mode (local/remote)      : Off / Off
Poll (local/remote)             : Off / Off
Tx Interval (local/remote)      : 100 / 100 ms
Rx Interval (local/remote)      : 100 / 100 ms
oper Tx Interval                : 100 ms
oper Rx Interval                : 100 ms
Multiplier (local/remote)       : 3 / 3
Local Diag                      : 0 (No Diagnostic)
Remote Diag                     : 0 (No Diagnostic)
Authentication                  : None
Clients                         : OSPF
Uptime                          : 00 days 04 hours 00 minutes 13 seconds
Up Count                        : 4
Last Valid Packet Rx            : 16:23:11.756480
Last Packet Tx                  : 16:23:11.730150
Photo of Sergey Vekli

Sergey Vekli

  • 506 Points 500 badge 2x thumb
What does mean "BFD path down"?
Photo of Sergey Vekli

Sergey Vekli

  • 506 Points 500 badge 2x thumb
Ok. We enable log for bfd events.
Photo of Henrique

Henrique, Employee

  • 10,302 Points 10k badge 2x thumb
Did you increase the BFD timer?
Photo of Sergey Vekli

Sergey Vekli

  • 506 Points 500 badge 2x thumb
No. Why we need to increase?
Photo of Henrique

Henrique, Employee

  • 10,302 Points 10k badge 2x thumb
It's just a suggestion to check if this flapping stops...
Photo of Sergey Vekli

Sergey Vekli

  • 506 Points 500 badge 2x thumb
After disabling BGP problem is disapear.