topic RE: Traceroute through MPLS cloud in ExtremeSwitching (EXOS/Switch Engine)

Traceroute through MPLS cloud

LEM — Thu, 06 Oct 2016 08:56:00 GMT

Traceroute through MPLS cloud.I have a simple demo stand for MP-BG. Here is how it looks:
CE1-PE1-PE2-CE2

We have BGP peering between CE and PE routers and OSPF, MPLS, MP-BGP inside provider cloud.

Both CE routers successfuly exhange routes. CE1 can see CE2 routes and vice versa.

I try to make trace from CE1 (192.168.101./0 ) to CE2 (192.168.102.0/24). Here is an output:

SW1.8 # traceroute 192.168.102.9 from 192.168.101.1
traceroute to 192.168.102.9, 30 hops max
1 172.16.13.3 1 ms 4 ms 13 ms
2 * * *
3 172.16.49.9 1 ms * 10 ms

Why do we have this string in output?
2 * * *

Here is a quoute from user guide:
In this mode, the MPLS TTL is independent of the IP TTL. The MPLS TTL is set to 255 on all packets
originated by the switch and on all packets that enter a pseudowire.

So, we should not see mpls path in trace.

RE: Traceroute through MPLS cloud

Erik_Auerswald — Thu, 06 Oct 2016 13:12:00 GMT

Hi,

traceroute prints asterisks (*) when it does not receive a TTL Exceeded message for a packet. A common reason for not receiving those are firewalls in layer 3 (routed) mode configured to not create ICMP messages.

Erik

RE: Traceroute through MPLS cloud

LEM — Thu, 06 Oct 2016 13:27:00 GMT

Hi, Erik!In my case, there are no firewalls/ACL.
I think that second response should be from end device. But there are three hops.

RE: Traceroute through MPLS cloud

Prashanth_KG — Thu, 06 Oct 2016 14:01:00 GMT

Hi,

Between the PE devices, do we have pseudowire for the transport of the CE traffic or we have MPLS routing in place?

Another possible reason in addition to Erik's suggestion:
If router does not respond within a timeout then traceroute prints an asterisk.

Hope this helps!

RE: Traceroute through MPLS cloud

LEM — Thu, 06 Oct 2016 14:16:00 GMT

There is MPLS routing between PE devices, not VPLS.

Is there any way to increase response time timeout? But i don't think it can be a problem, because those devices are in one rack.

RE: Traceroute through MPLS cloud

LEM — Fri, 07 Oct 2016 07:46:00 GMT

Any ideas?

RE: Traceroute through MPLS cloud

Erik_Auerswald — Fri, 07 Oct 2016 12:39:00 GMT

Hi,

can you tell us which devices can be seen in the trace? Is this the complete trace? Are the IP addresses from the incoming interfaces on PE1 and CE2?

Could it be that PE2 creates an ICMP TTL Exceeded message, but lacks a route to 192.168.102.9?

Can you take packet captures on the involved interfaces? That would allow you to see what really happens.

Erik

RE: Traceroute through MPLS cloud

Erik_Auerswald — Fri, 07 Oct 2016 12:39:00 GMT

I meant to write "192.168.101.1" instead of "192.168.102.9"...

RE: Traceroute through MPLS cloud

Prashanth_KG — Fri, 07 Oct 2016 13:24:00 GMT

Hi,

* represents no response from the specific hop.

I checked in the lab and even if we use the MPLS for the L3 routing, those hops are displayed.

Since we are doing a traceroute from a specific source-address, does all the routers in the path knows how to reach back to this source, 192.168.101.1?
I am suspecting there could be an asymmetric routing in the path.

Please try to do a ping to the destination from the source-address with a record-route.
ping 192.168.102.9 from 192.168.101.1 with record-route More details about the record-route can be fetched from the below article.
https://gtacknowledge.extremenetworks.com/articles/Q_A/How-is-PING-with-record-route-option-better-than-tracerouteHope this helps to narrow down the reason for the reported behaviour!

RE: Traceroute through MPLS cloud

LEM — Fri, 07 Oct 2016 13:24:00 GMT

It is vpnv4 scenario, so transit routes shoudn't know route to source. I will chek it with record-route

RE: Traceroute through MPLS cloud

LEM — Fri, 07 Oct 2016 13:59:00 GMT

Here is a ping from CE1:
* SW1.21 # ping 192.168.102.9 with record-route
Ping(ICMP) 192.168.102.9: 4 packets, 8 data bytes, interval 1 second(s).
16 bytes from 192.168.102.9: icmp_seq=0 ttl=253 time=8.635 ms
RR: 172.16.13.1 (CE1 interface)
172.16.13.3 (PE1-CE1 interface)
172.16.49.4 (PE2-CE2 interface)
192.168.102.9 (CE2 users interface)
172.16.49.9 (PE2-CE2 interface)
172.16.49.4
172.16.13.3
172.16.13.1

It looks like that 172.16.49.4 doesn't have route to 172.16.13.1. But look at iproute table:
* (vr CUSTOMER1-SITE2) SW4.83 # sh iproute
Ori Destination Gateway Mtr Flags VLAN Duration
#bi 172.16.13.0/24 10.100.34.3 0 UG-D---um-lf3 ISL-34 0d:0h:17m:43s
#d 172.16.49.0/24 172.16.49.4 1 U------um--f- ISL-49 0d:0h:19m:5s
#bi 192.168.101.0/24 10.100.34.3 1 UG-D---um-lf3 ISL-34 0d:0h:17m:43s
#be 192.168.102.0/24 172.16.49.9 0 UG-D---um--f- ISL-49 0d:0h:18m:22s

RE: Traceroute through MPLS cloud

Prashanth_KG — Fri, 07 Oct 2016 14:14:00 GMT

Hi,

Thank you for sharing the outputs.
On the customer1-site2 switch, can you please collect the output of

"rtlookup 172.16.13.1"

This will display the actual route being currently used by this switch for the destination?

Do we have the same traceroute output with both the commands below?

- traceroute 192.168.102.9
- traceroute 192.168.102.9 from 192.168.101.1

If there is a difference, please also collect the below output:
ping 192.168.102.9 from 192.168.101.1 with record-route Thanks in advance!

RE: Traceroute through MPLS cloud

LEM — Fri, 07 Oct 2016 14:17:00 GMT

Here is an ouput: * (vr CUSTOMER1-SITE2) SW4.83 # rtlookup 172.16.13.1 Ori Destination LSP EndPt Mtr Flags MPLS Label VLAN #bi 172.16.13.0/24 10.1.1.3/32 0 UG-D---um-lf3 0x00434 ISL-34 Traceroute path is same, because there is only 1 path there  * SW1.25 # ping 192.168.102.9 from 192.168.101.1 with record-route Ping(ICMP) 192.168.102.9: 4 packets, 8 data bytes, interval 1 second(s). 16 bytes from 192.168.102.9: icmp_seq=0 ttl=253 time=2.193 ms RR: 192.168.101.1 172.16.13.3 172.16.49.4 192.168.102.9 172.16.49.9 172.16.49.4 172.16.13.3 192.168.101.1 16 bytes from 192.168.102.9: icmp_seq=1 ttl=253 time=11 ms (same route) 16 bytes from 192.168.102.9: icmp_seq=2 ttl=253 time=12 ms (same route) 16 bytes from 192.168.102.9: icmp_seq=3 ttl=253 time=9.748 ms (same route) --- 192.168.102.9 ping statistics --- 4 packets transmitted, 4 packets received, 0% loss round-trip min/avg/max = 2/8/12 ms

RE: Traceroute through MPLS cloud

LEM — Fri, 07 Oct 2016 14:45:00 GMT

Also, i found something i can't uderstand. SW3 SW3.9 # sh bgp routes vpnv4 all Routes: Destination Label LPref Weight MED Peer Next-Hop AS-Path ------------------------------------------------------------------------------------ * i 65102:2::172.16.49.0/24 0x00074 100 1 0 10.1.1.4 10.1.1.4 * i 65102:2::192.168.102.0/24 0x00074 100 1 0 10.1.1.4 10.1.1.4 65102 Next hop to 192.168.102.0/24 goes via 10.1.1.4 SW3.10 # rtlookup 10.1.1.4 Ori Destination LSP EndPt Mtr Flags MPLS Label VLAN #mp 10.1.1.4/32 10.1.1.4/32 1 U--D---um-Lf- 0x00434 ISL-34 I ithink that #mp means that mpls should be used. But if look up route from vpnv4-vrf in SW * SW3.11 # * SW3.11 # vr "CUSTOMER1-SITE1" * (vr CUSTOMER1-SITE1) SW3.12 # sh iproute Ori Destination Gateway Mtr Flags VLAN Duration #d 172.16.13.0/24 172.16.13.3 1 U------um--f- ISL-13 0d:1h:1m:44s #bi 172.16.49.0/24 10.100.34.4 0 UG-D---um-lf3 ISL-34 0d:1h:0m:0s #be 192.168.101.0/24 172.16.13.1 1 UG-D---um--f- ISL-13 0d:1h:0m:18s #bi 192.168.102.0/24 10.100.34.4 1 UG-D---um-lf3 ISL-34 0d:1h:0m:0s Next hop to 192.168.102.0/24 is 10.100.34.4. It is also SW4 interface (SW4=PE2) It is strange that next-hop is different. Also, if i check global vrf path route to 10.100.34.4 there is no #mp flags: * SW3.14 # rtlookup 10.100.34.4 Ori Destination Gateway Mtr Flags VLAN Duration #d 10.100.34.0/24 10.100.34.3 1 U------um--f- ISL-34 0d:1h:5m:4s May be it can be a root of problem?

RE: Traceroute through MPLS cloud

Stephane_Grosj1 — Sun, 09 Oct 2016 01:16:00 GMT

Hi, That would require to open a case imho, as I'm wondering if this is an ICMP bug or not. The documentation would also need to be checked as it might refer to a previous gen of ASIC. I'm not sure of it, so investigation would be beneficial. My 2 cents.

RE: Traceroute through MPLS cloud

LEM — Mon, 10 Oct 2016 07:48:00 GMT

Well, i will try to open a case in TAC.
Thanks!

RE: Traceroute through MPLS cloud

Necheporenko__N — Mon, 10 Oct 2016 12:24:00 GMT

Hello LEM,

This are known issues documented under
CR xos0051847 - OAM: Traceroute via MPLS path does not display the intermediate hops
and
CR xos0016965 - EXOS kernel does not properly send back ICMP responses.

Currently we have no target release dates for them.

Best Regards,
Nikolay

RE: Traceroute through MPLS cloud

LEM — Mon, 10 Oct 2016 12:47:00 GMT

Nikolay, thanks!