Which interface are you using for the 40G connections?
Please collect the following from both switches with the 40G interfaces connected:
show fabric all
show fabric islports
show int fo
show media int fo
For the 40G connection to the ICX 7750, are you able to bring up the port-channel with just one interface?
With the 40G connected to the ICX, run:
show run int port-channel
show int port-channel
show port-channel summary
# check the lacp counters to see if the VDX is sending and receiving the LACP PDUs
From the above, we can check to see if there's anything obvious. If not, then you will need to collect a 'copy support-interactive' and open a case with Extreme to investigate further.