Extreme Networks

systemscsn · ‎08-20-2021

This started way before summer, and I thought it was fixed.

I made a minor change to the power on wifi1 interface (but basically any change i make), select the device, click on Update Device. Leave the default Update Network Policy and Configuration » Delta Configuration Update checked. I see the progress bar area show Queued for a few seconds, and then Device Update Failed is displayed next to the AP.

Hovering my pointer on the Device Failed to Update, i see this “Could not generate CLI configuration. Execute Method failed, with class”…………………….

Here are all the steps I tried, all of which failed:

I rebooted the AP, once it came back up, i tried a Delta Configuration Update. It failed.

I tried a Complete Configuration Update on the AP. It failed.

Rebooted the AP again, once it came back up I tried a Complete Configuration Update. It failed.

Went into the Aruba 2920 switch, disabled the port that the AP is plugged into (its like unplugging the AP), waited 10 minutes and then re-enabled the port. Once the AP came back up I tried a Delta Configuration Update. It failed.

Went back into the Aruba 2920 switch, disabled the same port again, waited 10 minutes and then re-enabled the port. Once the AP came back up I tried a Complete Configuration Update. It failed.

This is insane. This issue is going on with any of my 160 AP’s, doesn't matter what change I've made to the AP, whether its changing the channel its on, or the power on one of the interfaces. I remember when this first reared its ugly head, we just had the banner show up at the top of the web interface, stating it had been upgraded (IQ Engine?). We have not changed our network infrastructure in any way, so I'm absolutely positive its nothing we have done.

I need to make small changes to most the AP’s on campus. I need to fine tune all of our AP’s because of Cross Channel Interference, and so need to make changes with power and channels on the AP’s. This issue is holding me back.

How can this be resolved? We have 160 AP’s most of them are attached to the ceilings, so whatever the fix, it better not involve grabbing a paperclip, ladder and interrupting classes.

Whats going on? this is an AP650(AH), and all I did this time, was change the power from Auto to 5db.

I didnt realize my issue was still going on. But it never happened until an ExtremeIQ update (i remember seeing a banner ages ago, and thats when this started). The AP is running 10.0.r5.

Any help to get this resolved that does require a paperclip and ladder is appreciated.

Thanks,

J.

systemscsn · ‎09-27-2021

UPDATE and Possible Solution.

Let me start by saying the “fix” I was told to do, was to basically set the AP back to factory. After that, of course, id have to go back into it and give it a name, its original IP (no im not doing reservations), apply its network policy, and do a complete update. For 140+ AP’s that is not a very good fix, and quite honestly if other AP’s on the same firmware, plus the same model of AP were not failing, then of course this would work.

The reason I was given for the error, and something i’m very… VERY… upset about, was that when the backend was updated - im thinking multiple times - from Hive to ExtremeIQ, it messed something up on our end because of the version of firmware we are running. While i do believe it was something done on the backend (we had not made any changes on anything, network, AP, server related on our end) that for reasons i dont know, messed up most of our AP’s so they wouldn't do any kind of update, i dont think im convinced it was our firmware version. Because if that were true then all of our AP’s on that version would fail, and they are not. However im mad because if they went into updating the backend knowing it could mess up a customers AP’s, then an email to their customers telling them to upgrade to whatever version would have been appropriate, and i know I never received an email of a possible issue with their backend messing my AP’s up! To me, there was a basic lack of any kind of Quality Assurance when the backend was updated/migrated from Hive to XIQ and for that someone should take a spanking.

While they have escalated my support ticket, i have found a workaround. For any AP thats failing to Update, and has those errors about Bonjour. You MUST go into the AP, click on Configure and then select Bonjour Gateway Settings. Make sure you have the default value of 250 in the Priority Field (some of mine were either blank or had 10 as the value), Check the box for “Override the default realm name”, then Save the Bonjour Settings, Click on Update, Delta and then Update.

That seems to fix the issue. Of course, you can never uncheck that box, or the error will come back (and it doesn't matter what version of firmware running).

basically its getting that realm name from your Network 360 Plan, if you have created one, and placed the AP on its map. You wont see the update failed message if you have a value in Priority and dont have the AP within the Network 360 Plan.

This whole thing has been a pain, and will continue to be a pain until ive gone through every AP we have and make that change.

Once I've gone though that nightmare, i plan on upgrading the firmware to the latest version, even though i can only update them after hours - which is going to take an age to do -. To think this all came about from wanting to reduce the Transmission Power. Speaking of which, i did that on an AP that I made the above change to, and while it took the update, wifi0 is still showing auto instead of 5db, which i set it to. So, who knows, maybe checking the box gets rid of the error message, but doesn't update… really….. personally i think it just takes an age for the XIQ interface to reflect the changes on the AP, so i’m leaving it alone, and carrying on clicking!

Hope this helps the next poor bugger that may get this issue.

Ill post again if the person who got my escalated support ticket gets back to me with a better, less manual way to fix this problem.

Best,

Jason.

View solution in original post

systemscsn · ‎08-23-2021

Hi Ash,

yeah unfortunately, we have a ton… a ton of Macbooks and ipads as well as apple tv’s, and so its critical that “stuff” can be displayed, streamed, and printed using that nasty bonjour protocol. You are right though, it would show. Although i guess what i could do, would be almost a reverse and then what you suggest. I have our one-and-only spare AP. I could plug that in, make a change and see if it fails to update. if it does, i then create another policy to be just like the existing one (this will take some time) and then only apply the spare AP to it, and then see if it updates.

The issue has been passed on to our Extreme support contact, so we will see what he has to add (I shared this threads link, so he can see all the info, including whats not working, and things tried), and whether its something that's known on their side, and an easy fix, or whether they open a support case and have to escalate this to their dev teams.

So Ash, what do you think about the above, and the single spare AP?

Thanks again for giving me pointers.

J.

PS: Weird how your Realm text is below the check box, and mine is above… 😉

Ash_Finch · ‎08-23-2021

I think yes, likely it will fail regardless of whether it was WiFi0, or WiFi1 seeing as the error is mentioning to Bonjour.
Hm yeah if they’re all critical APs then moving to another policy may not be the best course of action, particularly if it was complicated to recreate, but it would 100% confirm Bonjour is to blame (or not)!

I would hope that adding an extra character to the realm name would be able to be reverted with the tickbox, but was unable to verify as mine look slightly different, but that may be because we don’t have it enabled:

43ee4caae8ae471d9c12af5af6103281_715123bd-733e-42ca-b291-420c1f1857cc.png

Have you raised a support case also?

systemscsn · ‎08-23-2021

Maybe I should have put it right in the first post, but just to be clear. I'm only making small changes to the wifi1 interface. I don't know if it will fail if i make a change on the wifi0 interface. Im thinking yes, due to the reason for it failing.

thanks,

systemscsn · ‎08-23-2021

Good morning Ash,

Yes, it has those two fields populated, heres a screen grab:

4b683e956ada4e0393f2b02acb1ef22d_12f303b1-3178-44fa-bd51-e36787f0392f.png

We havent changed anything network, server or otherwise, so its odd this started happening a few months back, and i thought it was resolved (i fixed the issue with another AP, simply by unplugging it from the network, and plugging it back in), but nothing ive tried has resolved this problem.

Now im wondering if it has something to do with the Realm Name being auto-generated.. I have most of my Network 360 building/floor plans done, drawn up, and AP’s placed. Im wondering if the auto-generated Real name has somehow got screwed up.

Would it case any issues (including bonjour - we rely heavily on that crap protocol) if i clicked on Override, and just put an “a” at the end of the realm name? and see if it delta updates? if it will cause an issue (stops bonjour working for even 5 minutes wouldnt be good) i wont be able to do that during hours.

Ive only ever added AP’s into XIQ, never taken one out…… but that would have to be done out of hours. Our network policy is quite complicated, so im quite concerned ill miss something if i create another one, and then only put a couple AP’s in it for testing. I just dont understand what happened thats causes this…. we literally haven't changed anything… the only changes would be that blue banner about an update applied (on extreme’s backend and the cloud based engine)…

Thanks, and i look forward to your reply, thanks for trying to help… i really need these buggers able to update.

Jason.

Ash_Finch · ‎08-23-2021

Not seen that error before...if you go to manage > devices > click on any of the hostnames (e.g. J-1) > configure > bonjour gateway settings > is this page populated with the priority and Realm Name fields?

A bit more along the lines of “have you tried turning it off and on again”, but have you tried removing an AP from XIQ and adding it back in again? Sometimes have had success with that.
I guess another thing to try temporarily is to create a new network policy and add just the SSIDs in (plus MGT/NATIVE VLANs and any DNS servers in additional settings also). If that works, then try adding the bonjour settings (newly created if possible, rather than the old object) and see if it likes it then or not.

Extreme Networks

Delta and Complete config update fails

Delta and Complete config update fails