Major Outage after trying to upgrade AP650 and 510C to 10.7.5.0

[systemscsn]

Hi,

Today I saw that 10.0.7.5 was out - being no 10.7.4.0 ever to came out - and all my AP's were on 10.7.3.0.  I selected a few that had zero clients on them - 4 of them - and three of them AP650(AH) went just fine while the AP510C failed with "Device Update Failed" and is showing as offline even though I can ping its IP address.

This is a MASSIVE ISSUE.  Because I when I saw the first 2-3 come up without issue, I went along and did BUNCH of AP's that didnt have clients and they all failed even though they are AP640(AH)'s.  They all have that message and are appearing offline

I know there was some big update to IQ Engine, I had that on the screen when i logged in today (been out for 4 days on vacation).  But I need to know how to get those AP's back online.  I have SSH'd into one of them and issues a reload command but so far its still showing that "Device Update Failed".  I even restarted the network switch, still nothing.  Of course XIQ showing it offline I cant do anything there, and thats why I tried SSH's in.

Any idea's would really help.  I need these **** things online as soon as I can, and Id also like to know why some failed and those others did NOT fail.  This is NOT good....  So if you know how I can get rid of that error and put these AP's back online, I would truly appreciate it.  Thanks, Jason.

[systemscsn]

I was just informed of a massive AWS outage, so it could be related.  BUT I also wan to add that I did notice at least two of the affected AP's have blinking white LED's and the other working ones are steady.  If I remember rightly that has something to do with PoE power.  Now, these are AP's that have been working without issue until I ran the update of firmware to 10.7.5.0, so it seems like its possibel there is a bug.  The only thing that's also very odd, is that I have a few of the same model AP's that took the new firmware and dint have any issues

Here is a shot of the AWS outage - we are in Florida - 

But I need to know when this is going to be resolved, or what steps I NEED to do. Or what the heck is the issue of an offline AP that I can ping, whose led is now blinking white when it never used to, and nothing has changed other than a firmware push.  HELP!!

[Macknkss]

Yeah this is scary. We did a delta update on our APs months ago and got the outage where Extreme did the DHCP update. Users unable to login for most of the day while we worked with support.

Good luck, hope you get it fixed ASAP. Good reminder to be cautious when doing any kind of update.

[systemscsn]

Thanks @Macknkss Extreme did the DHCP update?  Lost me.  They were doing the DHCP for you?  I have a DHCP/DNS server onsite, so at least I can rule that out.  Im starting to think its either AWS outage, or the firmware has a bug, or the timing of me sending the update and the AWS outage.  The reason being, is that in the last 10-20 minutes im up about 14 APs online.  I have 180 total, one offline because its a spare, but first thing this morning XIQ was showing only 2 AP's online... after a few hours it was back to 179, I did a test firmware push to ones I use for that, and all was going well, so then I targeted a bunch with no clients and thats when I saw the device update failed, blinking white LED and such.   I wish I knew if there was an SSH's command I could use inside the AP to do the whole "set device to template defaults" ... just as another test..

Thanks again, ill give it an hour or so, then I guess Ill have to open a GTAC ticket..... <sigh>..

Jason.

[Macknkss]

Sorry to confuse, my past issue is unrelated to yours for sure. They added this line to our AP configs during one of their updates that caused clients to not get IPs from our DHCP servers. "forwarding-engine dhcp-opt-82 enable"

We've had great luck with support so that would be the first thing I'd do. Good luck!