Traceroute through MPLS cloud

  • 0
  • 1
  • Question
  • Updated 2 years ago
  • Answered
Traceroute through MPLS cloud.I have a simple demo stand for MP-BG. Here is how it looks:
CE1-PE1-PE2-CE2

We have BGP peering between CE and PE routers and OSPF, MPLS, MP-BGP inside provider cloud. 

Both CE routers successfuly exhange routes.  CE1 can see CE2 routes and vice versa. 

I try to make trace from CE1 (192.168.101./0 ) to CE2 (192.168.102.0/24). Here is an output:

SW1.8 # traceroute 192.168.102.9 from 192.168.101.1
traceroute to 192.168.102.9, 30 hops max
 1  172.16.13.3                                  1 ms        4 ms       13 ms 
 2                                               *           *           *
 3  172.16.49.9                                  1 ms        *          10 ms

Why do we have this string in output?
 2                                               *           *           *

Here is a quoute from user guide:
In this mode, the MPLS TTL is independent of the IP TTL. The MPLS TTL is set to 255 on all packets
originated by the switch and on all packets that enter a pseudowire.

So, we should not see mpls path in trace.
Photo of LEM

LEM

  • 182 Points 100 badge 2x thumb

Posted 2 years ago

  • 0
  • 1
Photo of Erik Auerswald

Erik Auerswald, Embassador

  • 13,792 Points 10k badge 2x thumb
Hi,

traceroute prints asterisks (*) when it does not receive a TTL Exceeded message for a packet. A common reason for not receiving those are firewalls in layer 3 (routed) mode configured to not create ICMP messages.

Erik
Photo of LEM

LEM

  • 182 Points 100 badge 2x thumb
Hi, Erik!In my case, there are no firewalls/ACL.
I think that second response should be from end device. But there are three hops.
Photo of Prashanth KG

Prashanth KG, Employee

  • 5,300 Points 5k badge 2x thumb
Hi,

Between the PE devices, do we have pseudowire for the transport of the CE traffic or we have MPLS routing in place?

Another possible reason in addition to Erik's suggestion:
If router does not respond within a timeout then traceroute prints an asterisk.

Hope this helps! 
Photo of LEM

LEM

  • 182 Points 100 badge 2x thumb
There is MPLS routing  between PE devices, not VPLS.

Is there any way to increase response time timeout? But i don't think it can be a problem, because those devices are in one rack. 
Photo of LEM

LEM

  • 182 Points 100 badge 2x thumb
Any ideas?
Photo of Erik Auerswald

Erik Auerswald, Embassador

  • 13,792 Points 10k badge 2x thumb
Hi,

can you tell us which devices can be seen in the trace? Is this the complete trace? Are the IP addresses from the incoming interfaces on PE1 and CE2?

Could it be that PE2 creates an ICMP TTL Exceeded message, but lacks a route to 192.168.102.9?

Can you take packet captures on the involved interfaces? That would allow you to see what really happens.

Erik
Photo of Erik Auerswald

Erik Auerswald, Embassador

  • 13,792 Points 10k badge 2x thumb
I meant to write "192.168.101.1" instead of "192.168.102.9"...
Photo of Prashanth KG

Prashanth KG, Employee

  • 5,300 Points 5k badge 2x thumb
Hi,

* represents no response from the specific hop.

I checked in the lab and even if we use the MPLS for the L3 routing, those hops are displayed.

Since we are doing a traceroute from a specific source-address, does all the routers in the path knows how to reach back to this source, 192.168.101.1?
I am suspecting there could be an asymmetric routing in the path.

Please try to do a ping to the destination from the source-address with a record-route.
ping 192.168.102.9 from 192.168.101.1 with record-route  
More details about the record-route can be fetched from the below article.
https://gtacknowledge.extremenetworks.com/articles/Q_A/How-is-PING-with-record-route-option-better-than-traceroute
Hope this helps to narrow down the reason for the reported behaviour!
Photo of LEM

LEM

  • 182 Points 100 badge 2x thumb
It is vpnv4 scenario, so transit routes shoudn't know route to source. I will chek it with record-route
Photo of LEM

LEM

  • 182 Points 100 badge 2x thumb
Here is a ping from CE1:

* SW1.21 # ping 192.168.102.9 with record-route

Ping(ICMP) 192.168.102.9: 4 packets, 8 data bytes, interval 1 second(s).

16 bytes from 192.168.102.9: icmp_seq=0 ttl=253 time=8.635 ms

RR: 172.16.13.1 (CE1 interface)

172.16.13.3 (PE1-CE1 interface)

172.16.49.4 (PE2-CE2 interface)

192.168.102.9 (CE2 users interface)

172.16.49.9 (PE2-CE2 interface)

172.16.49.4

172.16.13.3

172.16.13.1


It looks like that 172.16.49.4 doesn't have route to 172.16.13.1. But look at iproute table:

* (vr CUSTOMER1-SITE2) SW4.83 # sh iproute

Ori Destination Gateway Mtr Flags VLAN Duration

#bi 172.16.13.0/24 10.100.34.3 0 UG-D---um-lf3 ISL-34 0d:0h:17m:43s

#d 172.16.49.0/24 172.16.49.4 1 U------um--f- ISL-49 0d:0h:19m:5s

#bi 192.168.101.0/24 10.100.34.3 1 UG-D---um-lf3 ISL-34 0d:0h:17m:43s

#be 192.168.102.0/24 172.16.49.9 0 UG-D---um--f- ISL-49 0d:0h:18m:22s
(Edited)
Photo of Prashanth KG

Prashanth KG, Employee

  • 5,300 Points 5k badge 2x thumb
Hi,

Thank you for sharing the outputs.
On the customer1-site2 switch, can you please collect the output of

"rtlookup 172.16.13.1"

This will display the actual route being currently used by this switch for the destination?

Do we have the same traceroute output with both the commands below?

- traceroute 192.168.102.9
- traceroute 192.168.102.9 from 192.168.101.1

If there is a difference, please also collect the below output:
ping 192.168.102.9 from 192.168.101.1 with record-route  
Thanks in advance!
Photo of LEM

LEM

  • 182 Points 100 badge 2x thumb
Here is an ouput:
* (vr CUSTOMER1-SITE2) SW4.83 # rtlookup 172.16.13.1
Ori Destination LSP EndPt Mtr Flags MPLS Label VLAN
#bi 172.16.13.0/24 10.1.1.3/32 0 UG-D---um-lf3 0x00434 ISL-34

Traceroute path is same, because there is only 1 path there :)

* SW1.25 # ping 192.168.102.9 from 192.168.101.1 with record-route
Ping(ICMP) 192.168.102.9: 4 packets, 8 data bytes, interval 1 second(s).
16 bytes from 192.168.102.9: icmp_seq=0 ttl=253 time=2.193 ms
RR: 192.168.101.1
172.16.13.3
172.16.49.4
192.168.102.9
172.16.49.9
172.16.49.4
172.16.13.3
192.168.101.1

16 bytes from 192.168.102.9: icmp_seq=1 ttl=253 time=11 ms
(same route)
16 bytes from 192.168.102.9: icmp_seq=2 ttl=253 time=12 ms
(same route)
16 bytes from 192.168.102.9: icmp_seq=3 ttl=253 time=9.748 ms
(same route)

--- 192.168.102.9 ping statistics ---
4 packets transmitted, 4 packets received, 0% loss
round-trip min/avg/max = 2/8/12 ms
Photo of LEM

LEM

  • 182 Points 100 badge 2x thumb
Also, i found something i can't uderstand.

SW3
SW3.9 # sh bgp routes vpnv4 all

Routes:
Destination Label LPref Weight MED
Peer Next-Hop AS-Path
------------------------------------------------------------------------------------
* i 65102:2::172.16.49.0/24 0x00074 100 1 0
10.1.1.4 10.1.1.4
* i 65102:2::192.168.102.0/24 0x00074 100 1 0
10.1.1.4 10.1.1.4 65102

Next hop to 192.168.102.0/24 goes via 10.1.1.4

SW3.10 # rtlookup 10.1.1.4
Ori Destination LSP EndPt Mtr Flags MPLS Label VLAN
#mp 10.1.1.4/32 10.1.1.4/32 1 U--D---um-Lf- 0x00434 ISL-34

I ithink that #mp means that mpls should be used.

But if look up route from vpnv4-vrf in SW
* SW3.11 #
* SW3.11 # vr "CUSTOMER1-SITE1"
* (vr CUSTOMER1-SITE1) SW3.12 # sh iproute
Ori Destination Gateway Mtr Flags VLAN Duration
#d 172.16.13.0/24 172.16.13.3 1 U------um--f- ISL-13 0d:1h:1m:44s
#bi 172.16.49.0/24 10.100.34.4 0 UG-D---um-lf3 ISL-34 0d:1h:0m:0s
#be 192.168.101.0/24 172.16.13.1 1 UG-D---um--f- ISL-13 0d:1h:0m:18s
#bi 192.168.102.0/24 10.100.34.4 1 UG-D---um-lf3 ISL-34 0d:1h:0m:0s

Next hop to 192.168.102.0/24 is 10.100.34.4. It is also SW4 interface (SW4=PE2)

It is strange that next-hop is different. Also, if i check global vrf path route to 10.100.34.4 there is no #mp flags:
* SW3.14 # rtlookup 10.100.34.4
Ori Destination Gateway Mtr Flags VLAN Duration
#d 10.100.34.0/24 10.100.34.3 1 U------um--f- ISL-34 0d:1h:5m:4s

May be it can be a root of problem?
Photo of Grosjean, Stephane

Grosjean, Stephane, Employee

  • 13,676 Points 10k badge 2x thumb
Hi,

That would require to open a case imho, as I'm wondering if this is an ICMP bug or not. The documentation would also need to be checked as it might refer to a previous gen of ASIC. I'm not sure of it, so investigation would be beneficial.

My 2 cents.
Photo of LEM

LEM

  • 182 Points 100 badge 2x thumb
Well, i will try to open a case in TAC.
Thanks!
Photo of Necheporenko, Nikolay

Necheporenko, Nikolay, Employee

  • 1,600 Points 1k badge 2x thumb
Hello LEM,

This are known issues documented under
CR xos0051847 - OAM: Traceroute via MPLS path does not display the intermediate hops
and
CR xos0016965 - EXOS kernel does not properly send back ICMP responses.

Currently we have no target release dates for them. 

Best Regards,
Nikolay
Photo of LEM

LEM

  • 182 Points 100 badge 2x thumb
Nikolay, thanks!