We ended up getting professional services involved to take a look at the issue.
Looking back through my notes, the reason for the problem is because is because our client certificate could not fit in one packet so was being fragmented across multiple packets. This is something LSNAT couldn't deal with at the time, so fragments were being dropped causing the TLS conversation to fail.
There was a bug fix in the S series firmware v8.31.01.005 which sounds like a similar issue:
"Fragmented packets are not allowed to traverse across an LSNAT6/4 or LSNAT4/6 vserver, the packets will be dropped"
In our case we were using straight LSNAT IPv4 to IPv4 with no IPv4/6 or IPv6/4 conversion.
I've never tried it again since the fix, might give it a go if I have time.
What hardware are you running and what firmware version are you on?
We did try a few things to reduce the size of the EAP packets, but from what i remember our client certificate was just too big.
Here are a few tips on that which were relevant when we had the issue with our NAC version - I would advise treading carefully, lots of potential to break stuff . Use wireshark/tcpdump on both client side and NAC side to monitor how the packets appear before and after LSNAT.
I. To reduce fragmentation on NAC:
Add the following appliance or appliance group properties and then enforce the appliance:
RADIUS_EAP_TLS_FRAGMENT_SIZE=1200
RADIUS_INNER_EAP_TLS_FRAGMENT_SIZE=1024
This will reconfigure the eap.conf and inner eap configuration files
II. To reduce packet size from client:
Microsoft’s KB on the subject (http://support.microsoft.com/kb/883389😞
The Extensible Authentication Protocol (EAP) packets of the RADIUS server are large when some firewall programs drop the UDP fragments to help protect the network. Framed MTU is used with EAP authentication to notify the RADIUS server about the Maximum Transmission Unit (MTU) negotiation with the client. The RADIUS server communicates with the client, so that the RADIUS server does not send EAP messages that cannot be delivered over the network. The default attribute value of the framed MTU for the IAS server is 1,500. You can set the attribute to a minimum of 64 and a maximum of 1,500. To avoid the fragment issues, you can set the attribute value to 1,344.
Thanks,
Mark