x690 new logspam since upgrading to 32.7.2.19-patch1-32

[Keith9]

We have two x690 stacks at our core that were running an old version 22.1.  We upgraded to 22.6.1.4 and then 32.7.2.19 patch1-32.

Everythings working fine but theres these new logspam entries.

On the second core stack this is repeating:
05/01/2025 09:13:00.64 <Warn:Card.IPv4FIB.Warning> Slot-2: vrId 131074 dest 10.14.0.249:32 nexthop 192.168.102.100: Unable to add route to unit 0, rc Entry exists. Shadow problem.
05/01/2025 09:13:00.12 <Noti:Card.IPv4FIB.Notice> Slot-1: vrId 2 dest 10.14.0.249:32 nexthop 192.168.102.100: DEFIP add uninstalled: Entry exists (-8): cannot add - ref 1, ins 0
05/01/2025 09:13:00.12 <Warn:Card.IPv4FIB.Warning> Slot-1: vrId 131074 dest 10.14.0.249:32 nexthop 192.168.102.100: Unable to add route to unit 0, rc Entry exists. Shadow problem.
05/01/2025 09:12:50.63 <Noti:Card.IPv4FIB.Notice> Slot-2: vrId 2 dest 10.14.0.249:32 nexthop 192.168.102.100: DEFIP add uninstalled: Entry exists (-8): cannot add - ref 1, ins 0
05/01/2025 09:12:50.63 <Warn:Card.IPv4FIB.Warning> Slot-2: vrId 131074 dest 10.14.0.249:32 nexthop 192.168.102.100: Unable to add route to unit 0, rc Entry exists. Shadow problem.
05/01/2025 09:12:50.63 <Noti:Card.IPv4FIB.Notice> Slot-2: vrId 2 dest 10.14.0.249:32 nexthop 192.168.102.100: DEFIP add uninstalled: Entry exists (-8): cannot add - ref 1, ins 0
05/01/2025 09:12:50.63 <Warn:Card.IPv4FIB.Warning> Slot-2: vrId 131074 dest 10.14.0.249:32 nexthop 192.168.102.100: Unable to add route to unit 0, rc Entry exists. Shadow problem.
05/01/2025 09:12:50.63 <Noti:Card.IPv4FIB.Notice> Slot-2: vrId 2 dest 10.14.0.249:32 nexthop 192.168.102.100: DEFIP add uninstalled: Entry exists (-8): cannot add - ref 1, ins 0
05/01/2025 09:12:50.63 <Warn:Card.IPv4FIB.Warning> Slot-2: vrId 131074 dest 10.14.0.249:32 nexthop 192.168.102.100: Unable to add route to unit 0, rc Entry exists. Shadow problem.

On the first core stack this every hour:
05/01/2025 00:36:19.55 <Noti:Card.IPv4FIB.LPMRsvdUsed> Slot-2: IPv4 route not added to hardware. Reached configured # of IP Route reserved-entries. (Logged at most once per hour.)

This command doesn't seem to show an issue with number of IP routes.

how iproute reserved-entries statistics
|-----In HW Route Table-----| |--------In HW L3 Hash Table-------|
# Used Routes # IPv4 Hosts IPv4 IPv4 IPv6 IPv4 IPv6
Slot Type IPv4 IPv6 Local Remote Local Rem. Local MCast MCast
---- ------------------------------- ------- ------ ------ ------ ------ ------ ----- ------ ------
1 X690-48x-2q-4c 15 0 244 0 0 0 0 38 0
2 X690-48t-2q-4c 15 0 244 0 0 0 0 38 0

Theoretical maximum for each resource type:
X435 32 16 64 64 509 4096 509 * 2048 * 1024
4120 64 32 64 64 509 512 256 * 256 * 128
X440G2 480 240 512 512 1021 4096 1021 * 2048 * 1024
X620 480 240 512 512 1533 4096 1533 * 2048 * 1024
4220, 5320-24T-4X-XT 992 496 1024 1024 4096 4096 2048 * 2048 * 1024
5320 16-port, 5320 24-port except XT 8160 4080 8192 8192 14334 16384 8192 * 8192 * 4096
5320 48-port, 5420F 12256 6128 12285 12288 14334 16384 8192 * 8192 * 4096
5420M 12256 6128 12288 12288 28670 32768 16384 * 16384 * 8192
X460G2 12256 6128 12288 12288 40958 49152 24576 * 24576 * 12288
X450G2 16352 8176 16381 16384 22526 28672 14336 * 14336 * 7168
5520 16352 8176 16384 16384 41982 65536 18429 * 32768 * 16384
X460G2-16MP 16352 8176 16384 16384 49152 49152 24576 * 24576 * 12288
5720MW 16352 8176 16384 16384 61438 98304 24573 * 49152 * 24576
X465, X590, X690 16352 8176 16384 16384 79870 135168 24573 * 67584 * 33792
5720MXW 24544 12272 24576 24576 143358 163840 81920 * 81920 * 40960
X695 32736 16368 32768 32768 102398 147456 57341 * 73728 * 36864
7520, 7720 32736 16368 32768 32768 102398 147456 57341 * 73728 * 36864

Anyway going to ask GTAC but curious if theres any dead giveaways here.

[Stephane_Grosj1]

You shouldn't have this message. GTAC is the good way to go.

Check what "show iproute reserved-entries" returns. By default it should be 12,240. Maybe the config has been corrupted for this part during the upgrade.

[emily852paul · Accepted solution]

Hello,

You're right to reach out to GTAC, but here's an initial breakdown that might clarify what you're seeing and why these messages likely started after your upgrade.

1. "Entry exists. Shadow problem" / "DEFIP add uninstalled"
These log entries: Unable to add route to unit 0, rc Entry exists. Shadow problem.
DEFIP add uninstalled: Entry exists (-8): cannot add - ref 1, ins 0
Indicate that the switch is trying to program a route into hardware, but the route already exists — or it's out of sync between software and hardware (aka “shadow” problem). The “DEFIP add uninstalled” means the route was not installed into the FIB because a similar entry is already there, and internal references/flags didn’t match cleanly.

This is typically:

A route refresh bug, where FIB state is partially stale after an update or failover

A route being learned multiple ways (e.g., static + OSPF/BGP), or redistribution quirks

A VR instance mismatch, especially relevant since vrId 2 vs vrId 131074 are being mentioned

2. "IPv4 route not added to hardware. Reached configured # of IP Route reserved-entries."

IPv4 route not added to hardware. Reached configured # of IP Route reserved-entries.
is logged once per hour as a warning, meaning the switch hit its current reserved capacity (not actual maximum). However:

Your stats:

Slot 1 and 2: # Used Routes: 15 (IPv4)
# IPv4 Hosts: 244
... show you're well below the platform max (X690: 16,352 IPv4 routes).

So what's wrong?

This is likely due to a reserved resource profile misconfiguration. After upgrading to EXOS 32.x, some platforms began strictly enforcing resource profiles, even if your actual usage is low. You might have the default resource profile, which reserves a low number of route entries in hardware.

✅ Recommendations
Check active resource profile:

show forwarding configuration
Look at the L3 Route Entries reservation.

Switch to a more route-heavy profile, if needed:

configure forwarding resource-profile <profile>
EXOS has different profiles for route-heavy, host-heavy, etc. You may want:

configure forwarding resource-profile route-intensive
This requires a reboot.

Check for duplicated/static routes:

show iproute destination 10.14.0.249

Check if it's being added from multiple sources (e.g., static + OSPF/BGP)

Confirm it's in the expected VR

Clear stale hardware FIB entries (if needed):
If GTAC confirms a FIB desync bug, they may guide you to clear/relearn routes:

restart process ipv4

Best Regard

Emily

[Gabriel_G]

If this is an AI reply, it's not terrible.

I generally agree with the suspects of some route refresh issue, or learning a route multiple ways causing some problem as your L3 table doesn't appear full despite the log messages.

As others mentioned, GTAC can get to the bottom of the problem if there isn't some more obvious issue like a stale config and/or multiple routes to the same destination causing some conflict.

[Keith9]

This was resolved by changing settings to the frr_ospf plugin on two upstream Netgate 8200 routers running in CARP High Availability mode.  The switch was learning the same route from both at the same time and I guess previous EXOS did not have the ability to log this condition.