topic BGP Edge router suggestions? in Network Architecture & Design

BGP Edge router suggestions?

Frank — Wed, 08 Jul 2015 19:55:00 GMT

I think I may start approaching our 480s' boundaries.

We're a multihomed datacenter, connected to let's say 4 upstream providers
We have two providers each on two 480s.
We receive FULL Internet routes + default from all 4 providers
We receive/advertise both IPv4 AND IPv6.
I also advertise a subset of these routes (severely limited) to our two core 8800 switches - mainly so that outbound traffic which traverses the 8800s has a chance of hitting the right egress router (the aforementioned 480s)

I've already made sure to limit the V4 routes to 500,000 from our neighbor adverts, but once they hit that limit, the tear-down/re-establishment of the BGP neighbor session doesn't help matters.
I already have route compression on, as well as "configure forwarding external-tables l3-only ipv4-and-ipv6" and "configure iproute reserved-entries maximum"

BGP process load can shoot up to 95%+ and in extreme cases, I think it's what makes a router reboot occasionally. Not fun! The only way I found to avoid that was to ditch a few routes from that neighbor to drop them to under-500,000. Still playing with and adjusting NLRI based policies. As I have default routes, I can afford to miss a few "real life" routes, but I'd really rather not.

So I'm thinking that we might need bigger boxes. What are my options?

- I want to have "Full Internet Routes plus default", meaning whatever the current mess is. I think it's over 512K (which is its own can of worms, I know) for V4 - and V6, but that's a lot fewer routes!
- I want to be able to support more than one upstream provider per box.
- My two boxes need to be able to talk to each other and exchange BGP routes properly
- If one or two upstream provider die unexpectedly and BGP routes gets seriously reshuffled, I don't want the box to fail.

Does Extreme have a bigger box for that? Or, if not, what would you suggest to get?

Thank you much for your input.

RE: BGP Edge router suggestions?

andreas1 — Wed, 08 Jul 2015 20:29:00 GMT

I do not think extreme has a box that big today. At least that do perform well. Are you looking for fast route updates and stability? What is your budget ?

RE: BGP Edge router suggestions?

Zelnosky__Kevin — Wed, 08 Jul 2015 20:33:00 GMT

Frank,

According to the 15.7 release notes supported limits, the only thing with bigger table sizes is the BlackDiamond X8 series with BlackDiamond X8-100G4X-XL modules.

The theoretical limit on these looks to be about 1,048,544 based on forwarding external table configuration "l3-only ipv4".

RE: BGP Edge router suggestions?

Frank — Wed, 08 Jul 2015 20:37:00 GMT

Mostly being able to hold all routes and being stable are my main goals. At this point I'm just trying to figure out what the entry point for such a router might be. "I'm a geek, not a budgeteer, Jim!" 🙂

RE: BGP Edge router suggestions?

Zelnosky__Kevin — Wed, 08 Jul 2015 20:47:00 GMT

Also, looks like we recently released a 40G12X-XL blade for the BD X8 that is not on the release notes yet. I'm not positive, but it should have the same table size as the 100G4X-XL.

RE: BGP Edge router suggestions?

Stephane_Grosj1 — Wed, 08 Jul 2015 23:14:00 GMT

Hi,

What EXOS release are you using?

With IP Route compression, the actual 540,000 routes are compressed to 280,000, but that's for the FIB. You can see it if you do a "sh iproute summary". IPv6 yields for ~20k more routes, and compress will reduce it significantly as well if you have it turned on (that's a different command than for IPv4).

If correctly set, you shouldn't have an issue with the FIB. Maybe the issue is more with the RIB, and thus the question about EXOS version, maybe you're running a non-optimal version in regards to BGP.

As Kevin pointed out, BDX8 has several blades with XL capacity, offering up to 1M entries in the FIB. The 40G-XL blade can act as a 10G blade as well through breakout cabling.

RE: BGP Edge router suggestions?

Zelnosky__Kevin — Wed, 08 Jul 2015 23:46:00 GMT

Opened a GTAC case for Frank to help figure out why BGP process is spiking so high as well as any issues with compression.

RE: BGP Edge router suggestions?

Manoharan__Sent — Mon, 27 Jul 2015 12:19:00 GMT

Hello,

We just want to few things about BGP process choke up to 95%.

Does this choke up happens frequently or during route learning/withdrawal?

The reason for asking this is BGP process memory goes up only during session formation & during route learning/withdrawal/convergence.After learning/peer formation process will settled down.
During heavy traffic BGP process should not choke up.
BCM RX only handle traffic utilization.
Moreover X480 is capable for handling 500K V4 routes from 2 peers.

If possible please check the logs during choke up.
In the meanwhile check whether convergence happening properly for route flapping/network
change & also check for route/network policy implemented properly.
Check also whether all routes learned to hardware through "sh iproute reserved-entries statistics".

We can assist our customer to raise case with their ISP & enable aggregation for some of the prefixes.Provider core have more powerful router, so they can add some aggregation entries.
Some missing prefixes will not have much impact since we already have default route to ISP.
By this way we can reduce the prefix to route table & reduce memory up to some extent.
Neglect this if customer had already did it.

Recent test with X8-XL cards with 500K internet routes doesn't shows any stability issues except some known convergence issues.

-Senthil.M

RE: BGP Edge router suggestions?

Frank — Mon, 27 Jul 2015 12:19:00 GMT

Senthil, you're correct - it only happens when we lose a neighbor - either due to maintenance or technical issues. It also doesn't always happen, which makes it really hard to replicate. The real issue is the ripple effect - if it happens, the router reboots, we've just lost two neighbors, and our customers experience a short time service loss, as BGP routes get recalculated.
Outside of those instances, the bgp process is pretty much idling along.

"sh iproute reserved-entries statistics" values seem to be well within limits.

I'm also experimenting with "what do I drop" - my current goal is to look at long AS path networks and see if I can filter those out in a reasonable way - that seems to me to be "fairer" than just dropping routes after receiving 500,000.

*sigh*, why doesn't the world just switch to IPv6 - this is only going to get worse! 😉

RE: BGP Edge router suggestions?

Manoharan__Sent — Mon, 27 Jul 2015 15:40:00 GMT

During router reboot check whether memory depletion happens.
From my understanding router reboot only during depletion of memory or when crash or watch dog expiry happens.

First we need to check the root cause of reboot/peer flap.
Since more routes are learned BGP process may choke up during route learning/convergence.
This should settle down after entire routes are learned & network is converged.
If not its serious issue.

From your point of view to discard routes through route policy in via "longest AS-PATH" is not possible.
The reason is current implementations for prefix deny/permit in policy is based on actual AS-NUMBERS not by number of AS-NUMBERS.
you may cross check using regular expressions.
In 16.2 (fBGP enhancement)only there is CLI for rejecting routes based on number of AS-PATHS.

You may reject/minimize prefixes by using "configure bgp neighbor maximum-prefix".
Advisable configuration is not to enable tear down.
Just enable max-prefix alone.

You may also configure dampening for route flapping.
Frequently flapping routes are penalized.

We suggest to configure maximum-prefix to ISP peers.

Any way we already have default route as savior for X480's to send the unknown destinations to provider edge routers.

Check these it may help.

-Senthil.M