BGP Edge router suggestions?

[Frank]

I think I may start approaching our 480s' boundaries.

We're a multihomed datacenter, connected to let's say 4 upstream providers
We have two providers each on two 480s.
We receive FULL Internet routes + default from all 4 providers
We receive/advertise both IPv4 AND IPv6.
I also advertise a subset of these routes (severely limited) to our two core 8800 switches - mainly so that outbound traffic which traverses the 8800s has a chance of hitting the right egress router (the aforementioned 480s)

I've already made sure to limit the V4 routes to 500,000 from our neighbor adverts, but once they hit that limit, the tear-down/re-establishment of the BGP neighbor session doesn't help matters.
I already have route compression on, as well as "configure forwarding external-tables l3-only ipv4-and-ipv6" and "configure iproute reserved-entries maximum"

BGP process load can shoot up to 95%+ and in extreme cases, I think it's what makes a router reboot occasionally. Not fun! The only way I found to avoid that was to ditch a few routes from that neighbor to drop them to under-500,000. Still playing with and adjusting NLRI based policies. As I have default routes, I can afford to miss a few "real life" routes, but I'd really rather not.

So I'm thinking that we might need bigger boxes. What are my options?

- I want to have "Full Internet Routes plus default", meaning whatever the current mess is. I think it's over 512K (which is its own can of worms, I know) for V4 - and V6, but that's a lot fewer routes!
- I want to be able to support more than one upstream provider per box.
- My two boxes need to be able to talk to each other and exchange BGP routes properly
- If one or two upstream provider die unexpectedly and BGP routes gets seriously reshuffled, I don't want the box to fail.

Does Extreme have a bigger box for that? Or, if not, what would you suggest to get?

Thank you much for your input.

[Manoharan__Sent]

During router reboot check whether memory depletion happens.
From my understanding router reboot only during depletion of memory or when crash or watch dog expiry happens.

First we need to check the root cause of reboot/peer flap.
Since more routes are learned BGP process may choke up during route learning/convergence.
This should settle down after entire routes are learned & network is converged.
If not its serious issue.

From your point of view to discard routes through route policy in via "longest AS-PATH" is not possible.
The reason is current implementations for prefix deny/permit in policy is based on actual AS-NUMBERS not by number of AS-NUMBERS.
you may cross check using regular expressions.
In 16.2 (fBGP enhancement)only there is CLI for rejecting routes based on number of AS-PATHS.

You may reject/minimize prefixes by using "configure bgp neighbor maximum-prefix".
Advisable configuration is not to enable tear down.
Just enable max-prefix alone.

You may also configure dampening for route flapping.
Frequently flapping routes are penalized.

We suggest to configure maximum-prefix to ISP peers.

Any way we already have default route as savior for X480's to send the unknown destinations to provider edge routers.

Check these it may help.

-Senthil.M

[Manoharan__Sent]

Hello,

We just want to few things about BGP process choke up to 95%.

Does this choke up happens frequently or during route learning/withdrawal?

The reason for asking this is BGP process memory goes up only during session formation & during route learning/withdrawal/convergence.After learning/peer formation process will settled down.
During heavy traffic BGP process should not choke up.
BCM RX only handle traffic utilization.
Moreover X480 is capable for handling 500K V4 routes from 2 peers.

If possible please check the logs during choke up.
In the meanwhile check whether convergence happening properly for route flapping/network
change & also check for route/network policy implemented properly.
Check also whether all routes learned to hardware through "sh iproute reserved-entries statistics".

We can assist our customer to raise case with their ISP & enable aggregation for some of the prefixes.Provider core have more powerful router, so they can add some aggregation entries.
Some missing prefixes will not have much impact since we already have default route to ISP.
By this way we can reduce the prefix to route table & reduce memory up to some extent.
Neglect this if customer had already did it.

Recent test with X8-XL cards with 500K internet routes doesn't shows any stability issues except some known convergence issues.

-Senthil.M

[Frank]

Senthil, you're correct - it only happens when we lose a neighbor - either due to maintenance or technical issues. It also doesn't always happen, which makes it really hard to replicate. The real issue is the ripple effect - if it happens, the router reboots, we've just lost two neighbors, and our customers experience a short time service loss, as BGP routes get recalculated.
Outside of those instances, the bgp process is pretty much idling along.

"sh iproute reserved-entries statistics" values seem to be well within limits.

I'm also experimenting with "what do I drop" - my current goal is to look at long AS path networks and see if I can filter those out in a reasonable way - that seems to me to be "fairer" than just dropping routes after receiving 500,000.

*sigh*, why doesn't the world just switch to IPv6 - this is only going to get worse! 😉

[Zelnosky__Kevin]

Opened a GTAC case for Frank to help figure out why BGP process is spiking so high as well as any issues with compression.