Problems with ipmcforward on Summitx690

[MDuRietz]

hi,
I'm experiencing some strange problems with multicast-routing.
I have an incoming multicast-trunk (untagged in on port) with a lot of different multicast-groups with different sources.
Most streams are working but some ar not. I cannot see what the problem is since nothing seems to differ in the streams (except for group and source-ips)
Can anyone help to see what i am missing?

System Type: X690-48x-2q-4c
Primary ver: 30.6.1.11
patch1-4


If i configure a mirror from the inconing vlan towards a host i can see the following in a packet-dump
------------------------------------------------------------------------------------------------------
Non-working stream:
13:44:46.321668 d0:c7:89:a8:37:c0 > 01:00:5e:10:09:c1, ethertype 802.1Q (0x8100), length 1374: vlan 2204, p 0, ethertype IPv4, (tos 0x88, ttl 59, id 23179, offset 0, flags [DF], proto UDP (17), length 1356) 10.225.1.193.1234 > 235.16.9.193.6000: [no cksum] UDP, length 1328
13:44:46.322044 d0:c7:89:a8:37:c0 > 01:00:5e:10:09:c1, ethertype 802.1Q (0x8100), length 1374: vlan 2204, p 0, ethertype IPv4, (tos 0x88, ttl 59, id 23180, offset 0, flags [DF], proto UDP (17), length 1356) 10.225.1.193.1234 > 235.16.9.193.6000: [no cksum] UDP, length 1328

Working stream:
13:44:50.857852 d0:c7:89:a8:37:c0 > 01:00:5e:10:00:ff, ethertype 802.1Q (0x8100), length 1374: vlan 2204, p 0, ethertype IPv4, (tos 0x88, ttl 60, id 0, offset 0, flags [DF], proto UDP (17), length 1356) 10.224.248.255.6000 > 235.16.0.255.6000: [no cksum] UDP, length 1328
13:44:50.858867 d0:c7:89:a8:37:c0 > 01:00:5e:10:00:ff, ethertype 802.1Q (0x8100), length 1374: vlan 2204, p 0, ethertype IPv4, (tos 0x88, ttl 60, id 0, offset 0, flags [DF], proto UDP (17), length 1356) 10.224.248.255.6000 > 235.16.0.255.6000: [no cksum] UDP, length 1328

TTL is ok, and the packets are pretty much identical but the switch refuses to route the first stream.

show vlan:
----------
Mcast_Transcoders 103 172.30.0.94 /27 -f------m-p---------------- ANY 5 /5 VR-Default
Mcast_In 2204 10.107.20.209 /26 -f------m-p---------------- ANY 2 /2 VR-Default


show iproute:
#s 10.224.248.0/24 10.107.20.193 1 UG---S-um--f- Mcast_In 61d:19h:7m:4s
#s 10.255.1.0/24 10.107.20.193 1 UG---S-um--f- Mcast_In 3d:22h:23m:48s

show pim:
---------
PIM Enabled, Version 2
PIM CRP Enabled on 3 interfaces
BSR state : ELECTED ; BSR Hash Mask : 255.255.255.252
Current BSR Info : 172.31.252.2 (Priority 10) expires after 1 sec
Configured BSR Info : 172.31.252.2 (Priority 10) in vlan Loopback-vlan
CRP Adv Interval : 60 sec ; CRP Holdtime: 150
BSR Interval : 60 sec ; BSR Timeout : 130
Cache Timer : 210 sec ; Prune Timer : 210
Assert Timeout : 210 sec ; Register Suppression Timeout,Probe: 60, 5
Generation Id : 0x611a4a32 Dense Neighbor Check : On
PIM-DM State Refresh TTL : 16
PIM-DM State Refresh Source Active Timer : 210
PIM-DM State Refresh Origination Interval : 60
Threshold for Last Hop Routers: 0 kbps
Threshold for RP : 0 kbps
Register-Rate-Limit-Interval : Always active
PIM SSM address range : None
PIM Register Policy : None
PIM Register Policy RP : None
PIM IP Route Sharing : Disabled
PIM IP Route Sharing Hash : Source-Group-Next Hop
Active Sparse Ckts 5 Dense Ckts 0 State Refresh Ckts 0

Global Packet Statistics ( In Out Drop )
C-RP-Advs 192710 192710 0
Registers 75943 75943 28
RegisterStops 75915 75915 0

VLAN Cid IP Address Designated Flags Hello J/P Nbrs
Router Int Int
Mcast_Transcoders 2 172.30.0.94 / 27 172.30.0.94 rifmsc------ 30 60 1
Loopback-vlan 5 172.31.252.2 / 32 172.31.252.2 rifms------- 30 60 0
Mcast_In 4 10.107.20.209 / 26 10.107.20.209 rifms---p-b- 30 60 0

show pim rp-set:
----------------
Group Mask C-RP Origin Priority Timeout
224.0.0.0 240.0.0.0 172.30.0.94 Bootstrap 0 109


sh pim cache (no source for the faulty stream):
-----------------------------------------------
[0000] 235.16.9.193 172.30.1.94 (WR) (null) Sparse

[0000] 235.16.0.255 172.30.0.94 (WR) (null) Sparse
[0001] 235.16.0.255 10.224.248.255 (S) Mcast_In Sparse


sh igpm snooping cache:
----------------------
pim 235.16.9.193 10.225.1.193 0 Mcast_In
Vlan Port Vid
Mcast_In 6 2204

pim 235.16.0.255 10.224.248.255 0 Mcast_In
Vlan Port Vid
Mcast_In 6 2204
Mcast_Transcoders 9 103
11 103
1 103
4 103
3 103

debugmode shows that the faulty stream has these log-entries (which the working ones does not):
-----------------------------------------------------------------------------------------------
03/25/2022 09:03:34.23 <Data:IPMC.Snoop.KernDelL2Cache> Deleting L2 Kernel Cache: src=10.225.1.193 group=235.16.9.193 inVlan=Mcast_In
03/25/2022 09:03:34.23 <Data:IPMC.VLAN.delCacheHal> Invoking HAL_IPv4_MC_DEL_CACHE src=10.225.1.193, grp=235.16.9.193, inVlan=Mcast_In, origVlan=1000171
03/25/2022 09:03:34.23 <Data:IPMC.Snoop.recvData> recv 52 bytes msgType = 5 from interface 1000171/1/6
03/25/2022 09:03:34.23 <Summ:IPMC.Snoop.l2CacheMiss> L2 Cache Miss src=10.225.1.193 group=235.16.9.193 interface=Mcast_In

does anyone have any suggestions on other filters for debug to see more information?
What might cause the deletion of L2 cache entry?

debug hal show ipv4Mc:
---------------------
vrId 2 G=235.16.0.225 S=10.224.248.225 Vid 2204 : (HW IPMC 11 l3hash 0 hit 0)
-> Vid 2204
-> 6
vrId 2 G=235.16.0.255 S=10.224.248.255 Vid 2204 : (HW IPMC 9 l3hash 0 hit 1)
-> Vid 103
-> 9 11 1 4 3
-> Vid 2204
-> 6
vrId 2 G=235.16.9.193 S=10.225.1.193 Vid 2204 : (HW IPMC 11 l3hash 0 hit 1)
-> Vid 2204
-> 6

Grateful for any input/suggestions on further troubleshooting

/Mattias

[SamPirok]

Happens to us all 😃 Thanks for updating the post with the solution!

[MDuRietz · Accepted solution]

Typical...It's funny how you go over the config over and over again and find nothing wrong, then suddenly you see it:

Wrong route: 10.255.1.0/24 should be 10.225....

Now i feel stupid