IS-IS MD5 authentication compute

Yoann_Jonard — Thu, 04 Dec 2025 10:04:46 GMT

Hello community,

I'm working on ISIS md5 authentication and I struggle to reproduce the digest the switches display.

I've read RFC 5304 which mentions that the Authentication Value field of the ISIS Hello PDU must be set to 0 before computing the authentication digest. Pretty simple isn'it ?

However, I am unable to reconstruct the digest the switches are sending to each other.

Here is the ISIS Hello PDU sent between a 5320 and a 4420 running VOSS 9.3.0.0 with MD5 authentication.

01deadbeefbb13001b00c00301080749c0de15152025f00f00000000cae4fd8c3d0484000000ca8102c18f8f7800000466000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000060c0080c201fd3c0080c202fd4c0a1136de92776eb44ac356fd0fbd934efcf233

The authentication value field is the last 16 bytes : de9277...
From my understanding I should only have to zeroed this field, and compute a HMAC-MD5 with the key ("totototototototo") and the ISIS PDU.
Note : The key has been configured to 16 bytes to exclude any kind of padding on the key as we can have with HMAC-SHA256.

This does not work.

As the RFC states "The HMAC-MD5 result for the IS-IS Hello PDUs SHALL be calculated after the packet is padded to the MTU size, if padding is not disabled", and the hello-padding is indeed enabled in the configuration.
Ok so let's pad to 1484 bytes (for a total frame size of 1513 bytes) with 0x00.
Why 1484 you would say ? Because that's the padding observed at the very beginning of the adjacency creation (which then disappear which suggests the padding is in loose mode)

It does not work

Let's pad with the same padding structure we observe in the capture : the padding tlv (08) + the padding length (max 254) + the padded zeros

It does not work.

Let's try all the possible padding length between 0 and 9600 (L3 MTU is different from L2 MTU but I'm kind of desperate here).

It does not work.

So I'm kind of lost here on why the digests do not match.

I've also tried to MD5 the key before applying the HMAC-MD5 but no luck. The only thing I can think of is that the key is somehow modified before the hmac computation, or another field of the PDU is zeroed.

Does someone know how this digest compute work and what I'm missing ?

Thank you,

Re: IS-IS MD5 authentication compute

flan — Wed, 17 Dec 2025 16:42:35 GMT

At first I was trying to figure out what you were trying to accomplish. If you are trying to figure out how they display the digest? I can't help you there. However, If you are talking about authenticating the NNI port. MD5 is not the best practice. The best practice is to use the hello-auth type hmac-sha-256. Give that a go and see if that works.

Flan

--See you in class!!!

Re: IS-IS MD5 authentication compute

Yoann_Jonard — Thu, 18 Dec 2025 09:13:38 GMT

Hello @flan ,

I'm trying to reproduce the digest observed on a packet capture. It seems simple, and it should be, but as you could read, I'm unable despite multiple permutation in the algorithm.

I know SHA-256 should be prefered to MD5, but I've also tried to reproduce the HMAC-SHA-256 digest without luck. The algorithm to compute the digest is also a bit more complex than MD5 so I decided to start as simple as possible with MD5.

Both authentication works on NNI links, I have no issue with this 🙂

topic Re: IS-IS MD5 authentication compute in ExtremeSwitching (VSP/Fabric Engine)

IS-IS MD5 authentication compute

Re: IS-IS MD5 authentication compute

Re: IS-IS MD5 authentication compute