Delta and Complete config update fails

[systemscsn]

This started way before summer, and I thought it was fixed.

I made a minor change to the power on wifi1 interface (but basically any change i make), select the device, click on Update Device.  Leave the default Update Network Policy and Configuration » Delta Configuration Update checked.  I see the progress bar area show Queued for a few seconds, and then Device Update Failed is displayed next to the AP.

Hovering my pointer on the Device Failed to Update, i see this “Could not generate CLI configuration.  Execute Method failed, with class”…………………….

 Here are all the steps I tried, all of which failed:

1. I rebooted the AP, once it came back up, i tried a Delta Configuration Update.  It failed.

2. I tried a Complete Configuration Update on the AP.  It failed.

3. Rebooted the AP again, once it came back up I tried a Complete Configuration Update. It failed.

4. Went into the Aruba 2920 switch, disabled the port that the AP is plugged into (its like unplugging the AP), waited 10 minutes and then re-enabled the port.  Once the AP came back up I tried a Delta Configuration Update.  It failed.

5. Went back into the Aruba 2920 switch, disabled the same port again, waited 10 minutes and then re-enabled the port.  Once the AP came back up I tried a Complete Configuration Update.  It failed.

This is insane.  This issue is going on with any of my 160 AP’s, doesn't matter what change I've made to the AP, whether its changing the channel its on, or the power on one of the interfaces.  I remember when this first reared its ugly head,  we just had the banner show up at the top of the web interface, stating it had been upgraded (IQ Engine?).  We have not changed our network infrastructure in any way, so I'm absolutely positive its nothing we have done.

I need to make small changes to most the AP’s on campus.  I need to fine tune all of our AP’s because of Cross Channel Interference, and so need to make changes with power and channels on the AP’s.  This issue is holding me back.

How can this be resolved?  We have 160 AP’s most of them are attached to the ceilings, so whatever the fix, it better not involve grabbing a paperclip, ladder and interrupting classes.

Whats going on?  this is an AP650(AH), and all I did this time, was change the power from Auto to 5db.

I didnt realize my issue was still going on.  But it never happened until an ExtremeIQ update (i remember seeing a banner ages ago, and thats when this started).  The AP is running 10.0.r5.

Any help to get this resolved that does require a paperclip and ladder is appreciated.

Thanks,

J.

[systemscsn]

BTW: The firmware I’m upgrading to, is the latest one, which is 10.3.4.0.  I’m even updating our AP1130’s that are running the old 8.2.4.0 version (some were on the 10.0.5.0).

Seems to be going well, although i did a separate post about how to use the distributed image upgrade option, and how it works.  If anyone knows how, I’m asking questions here: 

Ill be posting back here on this whole “Fix” I found for the update errors, as well as posting back to say whether this really fixes it, that i can not only update individual AP’s via Delta or Complete, but also that I see my changes within the XIQ interface, so I know it really is updating.

So I’m hoping this thread doesn't automatically close itself.

Thanks,

Jason.

[systemscsn · Accepted solution]

UPDATE and Possible Solution.

Let me start by saying the “fix” I was told to do, was to basically set the AP back to factory.  After that, of course, id have to go back into it and give it a name, its original IP (no im not doing reservations), apply its network policy, and do a complete update.  For 140+ AP’s that is not a very good fix, and quite honestly if other AP’s on the same firmware, plus the same model of AP were not failing, then of course this would work.

The reason I was given for the error, and something i’m very… VERY… upset about, was that when the backend was updated - im thinking multiple times - from Hive to ExtremeIQ, it messed something up on our end because of the version of firmware we are running.  While i do believe it was something done on the backend (we had not made any changes on anything, network, AP, server related on our end) that for reasons i dont know, messed up most of our AP’s so they wouldn't do any kind of update, i dont think im convinced it was our firmware version.  Because if that were true then all of our AP’s on that version would fail, and they are not.  However im mad because if they went into updating the backend knowing it could mess up a customers AP’s, then an email to their customers telling them to upgrade to whatever version would have been appropriate, and i know I never received an email of a possible issue with their backend messing my AP’s up!  To me, there was a basic lack of any kind of Quality Assurance when the backend was updated/migrated from Hive to XIQ and for that someone should take a spanking.

While they have escalated my support ticket, i have found a workaround.  For any AP thats failing to Update, and has those errors about Bonjour.  You MUST go into the AP, click on Configure and then select Bonjour Gateway Settings.  Make sure you have the default value of 250 in the Priority Field (some of mine were either blank or had 10 as the value), Check the box for “Override the default realm name”, then Save the Bonjour Settings, Click on Update, Delta and then Update.

That seems to fix the issue.  Of course, you can never uncheck that box, or the error will come back (and it doesn't matter what version of firmware running).

basically its getting that realm name from your Network 360 Plan, if you have created one, and placed the AP on its map.  You wont see the update failed message if you have a value in Priority and dont have the AP within the Network 360 Plan.

This whole thing has been a pain, and will continue to be a pain until ive gone through every AP we have and make that change.

Once I've gone though that nightmare, i plan on upgrading the firmware to the latest version, even though i can only update them after hours - which is going to take an age to do -.  To think this all came about from wanting to reduce the Transmission Power.  Speaking of which, i did that on an AP that I made the above change to, and while it took the update, wifi0 is still showing auto instead of 5db, which i set it to.  So, who knows, maybe checking the box gets rid of the error message, but doesn't update… really…..  personally i think it just takes an age for the XIQ interface to reflect the changes on the AP, so i’m leaving it alone, and carrying on clicking!

Hope this helps the next poor bugger that may get this issue.

Ill post again if the person who got my escalated support ticket gets back to me with a better, less manual way to fix this problem.

Best,

Jason.

[systemscsn]

UPDATE:

I opened a support ticket, and one of the first things they asked me to do was to update the firmware on an AP - one thats failing to update - from 10.0r5 to 10.3r3.

I updated the firmware and the Device update Failed didnt clean off the screen… odd.  but okay.

So i selected the AP, clicked on Update and then Delta… it failed.  I clicked on Update and Full Config… it failed….

I didnt think that would fix the issue, as i have so many more AP’s on the same firmware, some fail and others dont.

Now, just waiting on them to get back to me now that ive told them the above…. this sucks.

By rebooting the AP form the interface i can clear that failed message, but if i do any kind of update, it fails and the message comes up next to the device. 

its not great to see a staggering list of fails when in the ExtremeIQ Interface/Console.

so…...

[systemscsn]

So for the hell of it, i made a VERY minor change and thought to myself, “lets see who takes the delta and who doesnt”.  Bear in mind this is a VERY small selection in the screenshot below, but this is how it looks:

So NO reason to it at all.  Ones with or without an assigned location, different firmware, different device models, some fail and some do not.

I think ive reached the limit of asking questions and looking for help here.  Ill be create a support case tomorrow, and hope the fix it easy and can be done without affecting users - we are a school.

Totally sucks, and I'm a very disappointed customer at this point, especially given we haven't changed anything, so its not like we broke anything.

Thanks to the people that tried to help, its appreciated.

Jason.

[systemscsn]

Thanks Ash.  Im totally swamped, have been all week, but this issue has has gone on for a while with my hopes being the community would know.  ah well.  ll try get the ball rolling on this, but if i open it on friday, i wont be able to follow up much as we are out on monday and tuesday.

thanks,

J