topic RE: After upgrading to XMC 8.1.4.40, all AP's show as down? in ExtremeCloud IQ- Site Engine Management Center

After upgrading to XMC 8.1.4.40, all AP's show as down?

Steve_Ballantyn — Tue, 14 Aug 2018 20:17:00 GMT

Hello all,

I just ran an upgrade on my XMC server to version 8.1.4.40. When it came back up, it shows all of my wireless access points as being down. In other words, I have hundreds of "AP Out of Service" alarms. I tried a reboot of the XMC server, to no avail,

It is still connecting to my wireless controller and shows the AP's as active, and shows a number of clients that is accurate. But there is no longer any client history detail since the upgrade.

I am running 10.41.04.0002 on my C5210 controllers. Is there a mandatory upgrade on the wireless side of things?

RE: After upgrading to XMC 8.1.4.40, all AP's show as down?

Ryan_Yacobucci — Tue, 14 Aug 2018 20:26:00 GMT

Check the following:

in XMC go to administration --> diagnostics

Make sure the "Level" at the top is set to "Diagnostic"

Click on historical Statistics collection --> Collector status

Scroll down to the "Wireless Discover" section

Check the status of the SYNC

Does it say SYNC-SUCCEEDED or does it say INVALID SHARED SECRET?

If INVALID SHARED SECRET try the following:

SSH to the EWC and run the following commands:

secureconnection
message-bus-ciphers AES128-SHA256 3
apply

Give it about 10-15 minutes to attempt resync.

If it shows sync'd it'll need 30 minutes before everything shows back up normally.

Thanks
-Ryan

RE: After upgrading to XMC 8.1.4.40, all AP's show as down?

Steve_Ballantyn — Tue, 14 Aug 2018 20:26:00 GMT

Hello Ryan,

You are on the money - it's showing me a "SYNC_ERROR_INVALID_SHARED_SECRET" error.

I went onto my EWC and ran that command. I am not sure if it worked or not. I didn't get an error message, but I got this warning message ...

Warning: [AES128-SHA256] contains no NetSight client ciphers, one of the following:
AES128-SHA
RC4-SHA
RC4-MD5
Example to add NetSight client support: < message-bus-ciphers AES128-SHA256,AES128-SHA 3 >

RE: After upgrading to XMC 8.1.4.40, all AP's show as down?

Steve_Ballantyn — Tue, 14 Aug 2018 20:26:00 GMT

Hey, that fixed it Ryan. All of my false alarms vanished and my Wireless sync status is now showing "SYNC_SUCCEEDED".

Thank you very much!!

RE: After upgrading to XMC 8.1.4.40, all AP's show as down?

Ryan_Yacobucci — Tue, 14 Aug 2018 20:26:00 GMT

Check the "Collector Status" again to see if it says "SYNC-SUCCEEDED"

You may have to remove the weak ciphers that are listed.

If you SSH to the EWC and run the following:

secureconnection
show

What do you see?

RE: After upgrading to XMC 8.1.4.40, all AP's show as down?

Ryan_Yacobucci — Tue, 14 Aug 2018 20:26:00 GMT

Wonderful,

As a side note you'll want to make sure you remove the weak ciphers and disable the weak ciphers.

secureconnection
weak cipher disable

There was a change to support stronger ciphers for the EWC --> XMC connection. If anyone has previous had to enable weak ciphers on their EWC in the past they will need to perform this manual configuration to re-establish communication.

There will be an addendum put into the release notes regarding this change as currently the release notes indicate you need to change to a weak cipher, and this is not correct.

Thanks
-Ryan

RE: After upgrading to XMC 8.1.4.40, all AP's show as down?

Ronald_Dvorak — Wed, 15 Aug 2018 02:00:00 GMT

I've just upgraded my XMC to 8.1.4 and have the same issue.

Here my controller settings....

V2110-1:secureconnection# show
Weak Ciphers: disable

Message Cipher: none
V2110-1:secureconnection#

I don't see a way to change it via the GUI and that is a little weird for me because I can't think of anything that isn't possible via the GUI (beside deep troubleshooting via the shell).

RE: After upgrading to XMC 8.1.4.40, all AP's show as down?

Ryan_Yacobucci — Wed, 15 Aug 2018 02:04:00 GMT

Hey Ronald,

Correct, there is no way to change it in the GUI. You will need to add these commands by SSHing into the EWC. You don't have to shell all the way into the OS.

If your "Collector Status" shows the same "SYNC_ERROR_INVALID_SHARED_SECRET" message go ahead and run the commands:

secureconnection
message-bus-ciphers AES128-SHA256 3
apply

We should be releasing updated release notes with details soon.

Thanks
-Ryan

RE: After upgrading to XMC 8.1.4.40, all AP's show as down?

Ronald_Dvorak — Wed, 15 Aug 2018 02:04:00 GMT

Thanks working now.

RE: After upgrading to XMC 8.1.4.40, all AP's show as down?

Brian_Hively — Thu, 20 Sep 2018 01:53:00 GMT

I had the same issue and changing the message-bus-ciphers to AES128-SHA256 3 fixed the communication issues even without making a change to the weak-ciphers setting.

It appeared to be "factory defaulted" to weak-ciphers enabled and no message-bus-cipher configured prior to making this change to support XMC 8.1.4.40.

Thanks for posting the fix.

RE: After upgrading to XMC 8.1.4.40, all AP's show as down?

Dean_Ferraro — Thu, 25 Oct 2018 01:19:00 GMT

Great work Ron. Just came across this as we had recently upgraded to this version as well. Applied changes and all seems well. Dean

RE: After upgrading to XMC 8.1.4.40, all AP's show as down?

Marcus — Mon, 01 Jul 2019 19:44:00 GMT

Hi all,

I have the same problem "AP out of Service" in Extreme Management Center 8.1.5.22 with C35 Firmware Version 10.41.10.0014

I have check the status of Sync and it's ok "SYNC_SUCCEEDED"

Can someone help me?

Regards,
Marcus.