We typically do not certify compatibility of our optics with other vendors. I checked the Extreme Optics Compatibility chart for VDX and see that the 40G ER4 40km QSFP+ is supported on the VDX6940 and we recommend running the minimum version of NOS 7 when using this optic.
I did check to see if there were any known issues or reports of trouble when a VDX is connected to a Mellanox device. The only issues I could find were related to using 40G breakout cables where after connecting to a Mellanox switch the VDX interface will remain in “shutdown” until the VDX is reloaded. That issue was detailed in FN-2019-439.
Hopefully other users can confirm their working designs with VDX <> Mellanox.
Principal Technical Support Engineer / Extreme Networks