08-20-2021 02:37 PM
This started way before summer, and I thought it was fixed.
I made a minor change to the power on wifi1 interface (but basically any change i make), select the device, click on Update Device. Leave the default Update Network Policy and Configuration » Delta Configuration Update checked. I see the progress bar area show Queued for a few seconds, and then Device Update Failed is displayed next to the AP.
Hovering my pointer on the Device Failed to Update, i see this “Could not generate CLI configuration. Execute Method failed, with class”…………………….
Here are all the steps I tried, all of which failed:
This is insane. This issue is going on with any of my 160 AP’s, doesn't matter what change I've made to the AP, whether its changing the channel its on, or the power on one of the interfaces. I remember when this first reared its ugly head, we just had the banner show up at the top of the web interface, stating it had been upgraded (IQ Engine?). We have not changed our network infrastructure in any way, so I'm absolutely positive its nothing we have done.
I need to make small changes to most the AP’s on campus. I need to fine tune all of our AP’s because of Cross Channel Interference, and so need to make changes with power and channels on the AP’s. This issue is holding me back.
How can this be resolved? We have 160 AP’s most of them are attached to the ceilings, so whatever the fix, it better not involve grabbing a paperclip, ladder and interrupting classes.
Whats going on? this is an AP650(AH), and all I did this time, was change the power from Auto to 5db.
I didnt realize my issue was still going on. But it never happened until an ExtremeIQ update (i remember seeing a banner ages ago, and thats when this started). The AP is running 10.0.r5.
Any help to get this resolved that does require a paperclip and ladder is appreciated.
Thanks,
J.
Solved! Go to Solution.
09-27-2021 02:05 PM
UPDATE and Possible Solution.
Let me start by saying the “fix” I was told to do, was to basically set the AP back to factory. After that, of course, id have to go back into it and give it a name, its original IP (no im not doing reservations), apply its network policy, and do a complete update. For 140+ AP’s that is not a very good fix, and quite honestly if other AP’s on the same firmware, plus the same model of AP were not failing, then of course this would work.
The reason I was given for the error, and something i’m very… VERY… upset about, was that when the backend was updated - im thinking multiple times - from Hive to ExtremeIQ, it messed something up on our end because of the version of firmware we are running. While i do believe it was something done on the backend (we had not made any changes on anything, network, AP, server related on our end) that for reasons i dont know, messed up most of our AP’s so they wouldn't do any kind of update, i dont think im convinced it was our firmware version. Because if that were true then all of our AP’s on that version would fail, and they are not. However im mad because if they went into updating the backend knowing it could mess up a customers AP’s, then an email to their customers telling them to upgrade to whatever version would have been appropriate, and i know I never received an email of a possible issue with their backend messing my AP’s up! To me, there was a basic lack of any kind of Quality Assurance when the backend was updated/migrated from Hive to XIQ and for that someone should take a spanking.
While they have escalated my support ticket, i have found a workaround. For any AP thats failing to Update, and has those errors about Bonjour. You MUST go into the AP, click on Configure and then select Bonjour Gateway Settings. Make sure you have the default value of 250 in the Priority Field (some of mine were either blank or had 10 as the value), Check the box for “Override the default realm name”, then Save the Bonjour Settings, Click on Update, Delta and then Update.
That seems to fix the issue. Of course, you can never uncheck that box, or the error will come back (and it doesn't matter what version of firmware running).
basically its getting that realm name from your Network 360 Plan, if you have created one, and placed the AP on its map. You wont see the update failed message if you have a value in Priority and dont have the AP within the Network 360 Plan.
This whole thing has been a pain, and will continue to be a pain until ive gone through every AP we have and make that change.
Once I've gone though that nightmare, i plan on upgrading the firmware to the latest version, even though i can only update them after hours - which is going to take an age to do -. To think this all came about from wanting to reduce the Transmission Power. Speaking of which, i did that on an AP that I made the above change to, and while it took the update, wifi0 is still showing auto instead of 5db, which i set it to. So, who knows, maybe checking the box gets rid of the error message, but doesn't update… really….. personally i think it just takes an age for the XIQ interface to reflect the changes on the AP, so i’m leaving it alone, and carrying on clicking!
Hope this helps the next poor bugger that may get this issue.
Ill post again if the person who got my escalated support ticket gets back to me with a better, less manual way to fix this problem.
Best,
Jason.
08-30-2021 01:59 PM
I think we may need a support case to look in to this further, given the troubleshooting you’ve already done and the odd command lines you’re seeing in the configuration update. For the paperclip method, you are correct: holding it for less than 5 seconds is a reboot, more than 5 seconds is a factory reset. If the AP resets, you should see the LED blink off for a second or two, then come back on. It will not do anything different based on the kind of power source it is using.
08-30-2021 01:26 PM
the above was three days ago.
Nothing has changed, still cant push a delta because of those Bonjour issues. Heck, the text policy fails with the same errors, even though that policy doesn't have Bonjour enabled.
On the old production AP I can push the test policy to it without issues, and then push the original policy back to the AP WITHOUT an error.
That old production AP, is the same model as the ones having the issue…..
How does any of this make sense? I have basically done everything that can be done to test this, and yet it doesn't really pin-point the issue. Id like to say its Bonjour, but why then didn't pushing the original policy to that AP fail?
Question: If i use the paperclip method on one of the AP’s with this issue, but do it for less than 5 seconds, that reboots the AP, right? IBUT does it does more than if you pulled its PoE network cable out though?
Thanks,
J.
08-27-2021 11:22 AM
Well, this gets stranger, more complex and is a big issue at this point.
This morning when i came in, i went over to the building and pulled the network cables for that AP. Counted to 30, plugged them back in. When the AOP came back up, I went into it, and changed its wireless policy to the test policy, and pushed a Delta update.
It failed……………………
what's crazier, is i went into the audit for the AP, and this is what it shows:
The first screengrab shows that it the test policy was applied correctly but on the second screengrab shows that it failed with the same Bonjour errors, and the test policy does NOT have Bonjour configured, its disabled.
What the heck is going on…?
As you know, i was able to put the test policy on our spare (which was a production AP) AP - same model - and then put back the original network policy without any issues at all!!
Remember this isnt a case of a single AP, there is another on - same model - thats behaving exactly the same as this one…..
What now?
Regards,
Jason.
08-26-2021 05:11 PM
Hi Sam,
Well heres the thing, when i changed the test AP to use the Test Policy, I pushed a Delta Update, and the test AP didnt reboot. It did take the test policy, and was doing exactly what id set the test policy to do.
So im not sure why the light would stay solid white.
So first thing tomorrow, im going to get into one of the troublesome *im thinking its a heck of a lot more than the 2 i have now, that are acting up) AP’s, change it to the test policy, push a Delta update. Wait a few minutes and then im going to put it back to the original policy, and push the Delta update. See what it does, and if it fails, if it fails with the same error abotu Bonjour.
I just hope it doesn't really screw up that AP, as people will be wanting to connect to it by 8am (EST).
All of this is nuts though, the AP’s should be taking any Delta push im doing, based on the tiny changes i made.
Lets say it somehow magically fixed the issue with this AP, if I have to do this with the other 142 AP’s - same model - im not going to be a happy camper.
Thanks,
Jason.
08-26-2021 03:48 PM
Thanks for keeping us updated! I’d be interested to hear how your test policy goes with the troublesome AP. I can shed some light on the LED behavior- the LED will stay solid for 24 hours after a reboot, and then return to the blinking state. Complete configuration updates include a reboot of the device, so that would by why that LED is solid at the moment. Either way, solid or blinking, the white LED means the AP is connected to XIQ, so that’s good news at least.