After upgrading to XMC 8.1.4.40, all AP's show as down?

  • 0
  • 1
  • Problem
  • Updated 4 weeks ago
  • Solved
Hello all,

I just ran an upgrade on my XMC server to version 8.1.4.40. When it came back up, it shows all of my wireless access points as being down. In other words, I have hundreds of "AP Out of Service" alarms. I tried a reboot of the XMC server, to no avail,

It is still connecting to my wireless controller and shows the AP's as active, and shows a number of clients that is accurate. But there is no longer any client history detail since the upgrade.

I am running 10.41.04.0002 on my C5210 controllers. Is there a mandatory upgrade on the wireless side of things?

Photo of Steve Ballantyne

Steve Ballantyne

  • 5,806 Points 5k badge 2x thumb

Posted 2 months ago

  • 0
  • 1
Photo of Yacobucci, Ryan

Yacobucci, Ryan, Multi-Tier Technical Support Engineer

  • 5,470 Points 5k badge 2x thumb
Check the following: 

in XMC go to administration --> diagnostics 

Make sure the "Level" at the top is set to "Diagnostic"

Click on historical Statistics collection --> Collector status

Scroll down to the "Wireless Discover" section

Check the status of the SYNC

Does it say SYNC-SUCCEEDED or does it say INVALID SHARED SECRET?


If INVALID SHARED SECRET try the following: 

SSH to the EWC and run the following commands: 

secureconnection
message-bus-ciphers AES128-SHA256 3
apply


Give it about 10-15 minutes to attempt resync.

If it shows sync'd it'll need 30 minutes before everything shows back up normally.

Thanks
-Ryan
Photo of Steve Ballantyne

Steve Ballantyne

  • 5,806 Points 5k badge 2x thumb
Hello Ryan,

You are on the money - it's showing me a "SYNC_ERROR_INVALID_SHARED_SECRET" error.

I went onto my EWC and ran that command. I am not sure if it worked or not. I didn't get an error message, but I got this warning message ...

Warning: [AES128-SHA256] contains no NetSight client ciphers, one of the following:
AES128-SHA
RC4-SHA
RC4-MD5
Example to add NetSight client support: < message-bus-ciphers AES128-SHA256,AES128-SHA 3 >

Photo of Steve Ballantyne

Steve Ballantyne

  • 5,806 Points 5k badge 2x thumb
Hey, that fixed it Ryan. All of my false alarms vanished and my Wireless sync status is now showing "SYNC_SUCCEEDED".

Thank you very much!!
Photo of Yacobucci, Ryan

Yacobucci, Ryan, Multi-Tier Technical Support Engineer

  • 5,470 Points 5k badge 2x thumb
Check the "Collector Status" again to see if it says "SYNC-SUCCEEDED"

You may have to remove the weak ciphers that are listed. 

If you SSH to the EWC and run the following: 

secureconnection
show

What do you see?
Photo of Yacobucci, Ryan

Yacobucci, Ryan, Multi-Tier Technical Support Engineer

  • 5,470 Points 5k badge 2x thumb
Wonderful,

As a side note you'll want to make sure you remove the weak ciphers and disable the weak ciphers.

secureconnection
weak cipher disable


There was a change to support stronger ciphers for the EWC --> XMC connection. If anyone has previous had to enable weak ciphers on their EWC in the past they will need to perform this manual configuration to re-establish communication. 

There will be an addendum put into the release notes regarding this change as currently the release notes indicate you need to change to a weak cipher, and this is not correct.

Thanks
-Ryan
Photo of Ronald Dvorak

Ronald Dvorak, Embassador

  • 49,972 Points 20k badge 2x thumb
I've just upgraded my XMC to 8.1.4 and have the same issue.

Here my controller settings....

V2110-1:secureconnection# show
Weak Ciphers: disable

Message Cipher: none
V2110-1:secureconnection#





I don't see a way to change it via the GUI and that is a little weird for me because I can't think of anything that isn't possible via the GUI (beside deep troubleshooting via the shell).

Photo of Yacobucci, Ryan

Yacobucci, Ryan, Multi-Tier Technical Support Engineer

  • 5,470 Points 5k badge 2x thumb
Hey Ronald,

Correct, there is no way to change it in the GUI. You will need to add these commands by SSHing into the EWC. You don't have to shell all the way into the OS.

If your "Collector Status" shows the same "SYNC_ERROR_INVALID_SHARED_SECRET" message go ahead and run the commands:

secureconnection
message-bus-ciphers AES128-SHA256 3
apply


We should be releasing updated release notes with details soon.

Thanks
-Ryan
Photo of Ronald Dvorak

Ronald Dvorak, Embassador

  • 49,962 Points 20k badge 2x thumb
Thanks working now.
Photo of Brian Hively

Brian Hively

  • 80 Points 75 badge 2x thumb
I had the same issue and changing the message-bus-ciphers to AES128-SHA256 3 fixed the communication issues even without making a change to the weak-ciphers setting.

It appeared to be "factory defaulted" to weak-ciphers enabled and no message-bus-cipher configured prior to making this change to support XMC 8.1.4.40.

Thanks for posting the fix.