As Oscar said, the multicast table is rather limited on the x440 (G1). If you are not doing L3 on this switch, then change the lookup key for multicast.
By default (S,G,V) is used: it uses L3 Hash table.
I see you changed that to (*,G,V), which can help but:
1) you're still using the L3 hash table
2) it depends on your traffic type
So, the best way to scale up, if you don't do L3 Multicast (that includes PVLAN, MVR...), is to configure the lookup-key to mac-vlan.
configure forwarding ipmc lookup-key [group-vlan | source-group-vlan | mac-vlan | mixed-mode]
mac-vlan: Uses L2 Multicast FDB table with (DMAC, V) lookup. This is the default for Summit x430. This mode helps Multicast scaling on entry-level platforms where the L2 Multicast FDB table can store a significantly higher number of entries. As an example, Summit x440 can scale up to 4,000 entries in this mode compared to 192 (with Multicast compression enabled).
mixed-mode: Uses both L2 Multicast FDB and L3 Hash tables for multicast. In this mode, the following logic is applied on installing the cache entries in hardware:
- Multicast cache entries requiring to be forwarded across VLANs are installed in the L3 Hash table. This includes PIM, MVR and PVLAN cache entries.
- Multicast cache entries requiring L2 forwarding within a VLAN are installed in the L2 Multicast FDB table. This includes entries corresponding to IGMP Snooping, PIM Snooping and MLD Snooping.
- Any IPv4/IPv6 reserved multicast addresses (for example 224.0.0.x) are installed in the L3 Hash table as needed. These reserved addresses map to the following multicast MAC addresses: 01:00:5e:00:00:xx, 33:33:00:00:00:xx, 33:33:00:00:01:xx or 33:33:ff:xx:xx:xx.
Any change in the lookup key configuration causes all cache entries to be cleared, and traffic is temporarily dropped until the system re-learns the multicast caches and associated subscriptions.