Hardware route table full issues

[Paul_Thornton]

I've just been troubleshooting a rather odd network latency problem.

Scenario is that we have two X480s doing BGP with some Internet transit providers and peers. All fairly standard ISP setup - the transit providers give a default route, and the peers provide some more specifics - due to lack of room in the X480 for a full global IP routing table.

We were seeing latency jumps of over 100ms on all packets going through one of these switches earlier (which happened to have all of the BGP sessions for the peer routes on it). This increase in latency went away when these peers were shut down and the peer routes removed from the routing table. No processes were maxing out the CPU at the top of top though, so it didn't look like a classic slow-path issue.

The X480 should be able to cope with 256K IPv4 routes but only 8K IPv6 routes.

The BGP feeds were providing, just before I closed the peers, a total of around 81000 IPv4 routes and 26000 IPv6 routes.

With the extra peer routes removed, the total size of the routing table on the switch dropped to around 450 routes and everything was (and still is) happy again.

The switch log was complaining about:
01/26/2016 17:35:23.05 IPv6 route not added to hardware. Hardware LPM Table full.
which makes some sense given that 26K > 8K.

So, to the key part of the question. Assuming IPv6 traffic was very low (it was), should the overflow of the IPv6 hardware table have affected IPv4 forwarding? It very much seemed to have done so in this instance; we were testing with IPv4 traffic to devices on the network and all suffered increased latency.

And as a followup, is there a way to re-carve this on an X480 to give more IPv6 hardware entries?

Thanks

Paul.

[Paul_Thornton]

Perfect.
There's plenty of headroom there, given that we have no intention of putting a full v4 or v6 Internet routing table in the switch.

Thanks for the quick answer, Stephane.

Paul.

[Stephane_Grosj1]

To check, the command is show iproute reserved-entries statistics.

The meaning is that the RIB has 570982 entries, from which 287247 have been compressed, meaning the FIB has been provided with 570982 - 287247 = 283735 entries.

What you see with the command you typed (show iproute reserved-entries - you were not far from the right command ;)) is the number of entries reserved for LPM entries in the LPM HW table. Which mean the rest may be used by other entries (ARP...).

[Paul_Thornton]

Hi all,

A brief followup on this as I've been doing some lab testing today.

I have an X480 here, and I've reconfigured it to have l3-only ipv4-and-ipv6 forwarding configuration and ip route compression enabled. All went as expected.

So, in the interests of science, I sent it a full Internet routing table:

* lab_inet1.63 # show iproute summary
=================ROUTE SUMMARY=================
Mask distribution:
1 default routes 16 routes at length 8
13 routes at length 9 35 routes at length 10
101 routes at length 11 265 routes at length 12
507 routes at length 13 1029 routes at length 14
1764 routes at length 15 12944 routes at length 16
7346 routes at length 17 12372 routes at length 18
25189 routes at length 19 36581 routes at length 20
38614 routes at length 21 63736 routes at length 22
54674 routes at length 23 315613 routes at length 24
1 routes at length 25 1 routes at length 26
12 routes at length 27 20 routes at length 28
38 routes at length 29 59 routes at length 30
7 routes at length 31 44 routes at length 32

Route origin distribution:
570874 EBGP 2 Blackhole 31 Static
75 Direct

Total number of routes = 570982
Total number of compressed routes = 287247

My question here is do the last two lines mean it has 570K routes in total (it does, that's pretty much the size of the global routing table), of which 287K are compressed?

So am I actually using up 283K routes in the hardware table? Or 570K? Or somewhere in between? Is there a way to find out?

I see this, and no log entries, so as it has more than 458K routes the hardware isn't overfilled:

lab_inet1.65 # show iproute reserved-entries
IPv4 # Reserved Routes Minimum #
Slot Type Routes IPv4 (or IPv6) IPv4 Hosts
---- ---------------- -------- ------- ------------------ ----------
1 X480-24x(10G4X) External 458720 ( ext.) [default] 16384

but that doesn't change when I add routes to the switch.

As a benchmark, the X480 had no problems loading the full BGP table. It was over a WAN link, and took 6 minutes 40 seconds, with dcbgp near the top of top for the duration.

Paul.

[Stephane_Grosj1]

Yes, it's usually difficult to find the reason of an issue when troubleshooting a (now) working network. 🙂