Question

Delta and Complete config update fails


Userlevel 3

 

This started way before summer, and I thought it was fixed.

 

I made a minor change to the power on wifi1 interface (but basically any change i make), select the device, click on Update Device.  Leave the default Update Network Policy and Configuration » Delta Configuration Update checked.  I see the progress bar area show Queued for a few seconds, and then Device Update Failed is displayed next to the AP.

 

Hovering my pointer on the Device Failed to Update, i see this “Could not generate CLI configuration.  Execute Method failed, with class”…………………….

 

 Here are all the steps I tried, all of which failed:

  1. I rebooted the AP, once it came back up, i tried a Delta Configuration Update.  It failed.
  1. I tried a Complete Configuration Update on the AP.  It failed.
  1. Rebooted the AP again, once it came back up I tried a Complete Configuration Update. It failed.
  1. Went into the Aruba 2920 switch, disabled the port that the AP is plugged into (its like unplugging the AP), waited 10 minutes and then re-enabled the port.  Once the AP came back up I tried a Delta Configuration Update.  It failed.
  1. Went back into the Aruba 2920 switch, disabled the same port again, waited 10 minutes and then re-enabled the port.  Once the AP came back up I tried a Complete Configuration Update.  It failed.

 

This is insane.  This issue is going on with any of my 160 AP’s, doesn't matter what change I've made to the AP, whether its changing the channel its on, or the power on one of the interfaces.  I remember when this first reared its ugly head,  we just had the banner show up at the top of the web interface, stating it had been upgraded (IQ Engine?).  We have not changed our network infrastructure in any way, so I'm absolutely positive its nothing we have done.

I need to make small changes to most the AP’s on campus.  I need to fine tune all of our AP’s because of Cross Channel Interference, and so need to make changes with power and channels on the AP’s.  This issue is holding me back.

How can this be resolved?  We have 160 AP’s most of them are attached to the ceilings, so whatever the fix, it better not involve grabbing a paperclip, ladder and interrupting classes.

 

 

Whats going on?  this is an AP650(AH), and all I did this time, was change the power from Auto to 5db.

 

I didnt realize my issue was still going on.  But it never happened until an ExtremeIQ update (i remember seeing a banner ages ago, and thats when this started).  The AP is running 10.0.r5.

 

Any help to get this resolved that does require a paperclip and ladder is appreciated.

 

Thanks,

J.


40 replies

Hi J,

 

Last time this happened to me i had to reset the AP template within the network policy

I when into the AP template, clicked save and then went back to the managed devices screen as was able to update the AP via delta without any issue

Cheers

Shane

Userlevel 3

Thanks for this.  But just so i Understand.

 

Click on Configure, then Network Policy and click on the device template.

Once that comes onto the screen, just click Save down the bottom right.

After that, go back to manager » devices, select the one im trying to update, and then do the delta update?

 

Because if i do that, then im going to get that orange icon next to all the AP’s, and will have to do a delta update on all the AP’s.

let me know though, please.

Correct

Userlevel 3

Correct

K, Going to try that now.  Ill post back.

Userlevel 3

Okay, they all went orange.  I selected just the AP that keeps failing, tried the Delta… it failed.  it did take longer to fail about 40 seconds, but it failed..

 

<sigh>….

On the configuration missmatch next to the AP that you have just tried to update, if you click on this against either “delta”  it should give the entire reason as to why the update failed

Userlevel 6

Try the following on an AP:

 

1 - Upgrade to the latest OS version but make sure that the Update Network Policy and Configuration box is unchecked

2 - If the firmware upgrade is successful, push a Complete Config update. 

Userlevel 3

Hi Shane. Thanks for getting back to me.

 

In clicking that, it was strange.  I got this first:

 

I closed the box, refreshed the screen, clicked on it again and it didnt show that box saying something is already in progress.  Dont know how that would even be possible, as ive been away from my desk for the last 45 minutes!

After the refresh, and clicking on it, i got this:

 

This is probably due to be going into the device template and clicking on save.

Clicking on Audit, i get this message:

We are running Bonjour gateway via Extreme, and have one AP in each division set up (with different priorities, just in case the AP acting as the main BJ GW goes down.

 

Does this help figure this out?

 

Christoph S: No way im upgrading the firmware on all of these AP’s.  This is about the most stable versions there is, plus nothing changed on our side of things, not AP-wise, or infrastructure-wise.

 

Any ideas Shane?

 

thanks,

J

PS: I did a search of all these community pages, and nothing at all for either of those two things concerning bonjour, nothing….. UGH I despise bonjour/apple devices...

Userlevel 4

Not seen that error before...if you go to manage > devices > click on any of the hostnames (e.g. J-1) > configure > bonjour gateway settings > is this page populated with the priority and Realm Name fields?

A bit more along the lines of “have you tried turning it off and on again”, but have you tried removing an AP from XIQ and adding it back in again? Sometimes have had success with that.
I guess another thing to try temporarily is to create a new network policy and add just the SSIDs in (plus MGT/NATIVE VLANs and any DNS servers in additional settings also). If that works, then try adding the bonjour settings (newly created if possible, rather than the old object) and see if it likes it then or not.

Userlevel 3

Good morning Ash,

 

Yes, it has those two fields populated, heres a screen grab:

We havent changed anything network, server or otherwise, so its odd this started happening a few months back, and i thought it was resolved (i fixed the issue with another AP, simply by unplugging it from the network, and plugging it back in), but nothing ive tried has resolved this problem.

Now im wondering if it has something to do with the Realm Name being auto-generated.. I have most of my Network 360 building/floor plans done, drawn up, and AP’s placed.  Im wondering if the auto-generated Real name has somehow got screwed up.  

Would it case any issues (including bonjour - we rely heavily on that crap protocol) if i clicked on Override, and just put an “a” at the end of the realm name? and see if it delta updates?  if it will cause an issue (stops bonjour working for even 5 minutes wouldnt be good) i wont be able to do that during hours.

 

Ive only ever added AP’s into XIQ, never taken one out……  but that would have to be done out of hours.  Our network policy is quite complicated, so im quite concerned ill miss something if i create another one, and then only put a couple AP’s in it for testing.  I just dont understand what happened thats causes this…. we literally haven't changed anything… the only changes would be that blue banner about an update applied (on extreme’s backend and the cloud based engine)…

 

Thanks, and i look forward to your reply, thanks for trying to help… i really need these buggers able to update.

Jason.

Userlevel 3

Maybe I should have put it right in the first post, but just to be clear.  I'm only making small changes to the wifi1 interface.  I don't know if it will fail if i make a change on the wifi0 interface.  Im thinking yes, due to the reason for it failing.

 

thanks,

Userlevel 4

I think yes, likely it will fail regardless of whether it was WiFi0, or WiFi1 seeing as the error is mentioning to Bonjour.
Hm yeah if they’re all critical APs then moving to another policy may not be the best course of action, particularly if it was complicated to recreate, but it would 100% confirm Bonjour is to blame (or not)!

I would hope that adding an extra character to the realm name would be able to be reverted with the tickbox, but was unable to verify as mine look slightly different, but that may be because we don’t have it enabled:

 
 

Have you raised a support case also?
 

Userlevel 3

Hi Ash,

 

yeah unfortunately, we have a ton… a ton of Macbooks and ipads as well as apple tv’s, and so its critical that “stuff” can be displayed, streamed, and printed using that nasty bonjour protocol.  You are right though, it would show.  Although i guess what i could do, would be almost a reverse and then what you suggest.  I have our one-and-only spare AP.  I could plug that in, make a change and see if it fails to update.  if it does, i then create another policy to be just like the existing one (this will take some time) and then only apply the spare AP to it, and then see if it updates.

The issue has been passed on to our Extreme support contact, so we will see what he has to add (I shared this threads link, so he can see all the info, including whats not working, and things tried), and whether its something that's known on their side, and an easy fix, or whether they open a support case and have to escalate this to their dev teams.

 

So Ash, what do you think about the above, and the single spare AP?

 

Thanks again for giving me pointers.

J.

PS: Weird how your Realm text is below the check box, and mine is above… ;)

Userlevel 4

Yeah using a spare AP as a test would also be worth a go as you’ve mentioned, and may make it easier to factory reset also! It may not get you much further, but at least then support would be able to jump straight into the bonjour and work out what’s going on with that and gives them an AP to try stuff to!

Userlevel 3

Hi Ash,

 

Wish there was a way to duplicate a network policy, as doing that and just editing it so that people dont accidently attach to the AP that you apply the new policy to, would be great.  thats my worry.  I create the policy, put that spare AP into it, and people attach to that AP.  Then again, i can give it a different SSID and WPA2 on it, they wont be able to join/connect to it…

 

Now of course, its about finding time to do that.

 

This whole deal has been passed along to our Tech Rep with Extreme, and the link to this page as well.  So we will see.  But i still plan on updating this frequently, as i dig a bit more into this.  Being at my desk about 50% of the day, and the other 50% out and about taking care of work orders, I can try things in the interface and see if that resolves the issue.  First thing, would be for me to plug in that spare, make a change (same change - disable Auto power on wifi1, setting it to 5db), and then applying the delta and see if that one fail (it will fail… i was being optimistic).  I have that spare in storage here.  Next will be the policy.  I know that spare is already onboarded, so ill have to go into it and set it to use the other policy that ive got to create.  never done that before.  But i believe all I have to do, is go into the AP, select Configure » Device Configuration, and select the dropdown, and select the new policy i just created.  Sound right?

 

Thanks much, really appreciated,

J.

Userlevel 4

To be honest I guess the main thing is the bonjour piece, so you could just literally create a new network policy > add any SSID that’s WPA2 or something > push the update out. Just no bonjour.
I don’t think that would fail, but we’ll see!
Then add the bonjour and try and update again. I suspect it would fail at this point given what’s been said so far.

And yes, just go to the AP in that section change the policy or alternatively can be done via manage > tick AP > actions > assign network policy, select the new one and try to push it out.

Userlevel 3

Hi Ash,

 

yeah, i think thats whats tripping it up.  its just crazy that nothing has changed on our end.  Ive been scratching my head, and i know ive done delta config pushes without any issues, and I want to say this didnt start happening until around 3 banners ago.  By banners, i mean the one that shows up at the top of the interface after you login, and says there has been an update (im guessing on the backend stuff, such as features, etc.).

So, im wondering if something got broken on the back end of things.  We literally havent changed anything infrastructure-wise.

 

Thats a really good point, just create the policy with almost anything in it, without bonjour, push and see.  and how about this:

See if it fails or not (im with you, i don think it will fail) I change the AP putting it back in the original policy (which has bonjour) and then try pushing the config and see if it takes it or not.

Thoughts? 

 

OH, question.  If i create the new network policy, go through all the settings that I have to.  If i click on save instead of deploy (ill have to do this first thing tomorrow morning before kids show up), it just saves it and the temp SSID wont show up?  Im a little fragged with all the things ive had to to do today, a bit punchy, hence probably a dumb question.

To elaborate, the test SSID I just created inside the test network policy will only show up, once i go into the test AP, change its policy to my test one, and push the config update, right?

 

:)

Userlevel 4

Quite possible that something was broken in an update somehow. Would make sense anyway if it previously worked and you haven’t made any changes to bonjour since.
Yeah that method would make sense in proving it’s something in that policy.

As long as you don’t push a delta/complete to the AP, any changes in XIQ are just shown in XIQ. It won’t change the AP in any way. So yeah that’s correct that the test SSID will only show once the config update is pushed to the test AP.

Userlevel 3

Thanks Ash, appreciated.

While i did make it into work very early this morning, i had quite a few things to take care of, so I didnt get a chance to try it.  Id left the window open on creating a new network policy and it must have auto-saved, as I have the second one showing, which contains the basic policy.

 

I know there is a way to hide the SSID inside the policy, and have been debating if i want to do that.  Obviously no one is going to connect to the spare AP (after i push the config) as they wont know the password.

Sorry to keep bombarding you with question, thanks for being patient.  I was wondering something; i create that policy and name the SSID in that test policy as “Testing_Push”, i then push it to that AP.  Im I correct that the “Testing_Push” SSID will only be seen by a device thats close to that AP and not across all the AP’s?

Im hopeful that ill be able to finish off this test, and post back what happens.  I am curious what it will do after I apply the new policy, then put the original policy back.  My guess is the test policy will apply just fine, but will fail when i change it back to the original policy.

 

Thanks,

J

Userlevel 4

Hiding the SSID can be done via the optional setting at the bottom of the SSID page > tick Hide SSID (Stealth mode)
Correct. APs can only have one policy assigned. Therefore the test AP will just have that one policy and the test SSID. All other APs will be on the original policy and therefore won’t broadcast the test ssid. Thus, only when you’re near the test AP that you’ll see that SSID.

Userlevel 3

Thanks Ash.

That is very interesting.  So you could literally have each AP have its own SSID.  You could do SSID’s by location, by room….. Got to be a limit on the number of SSID’s you can create…. I suppose its nice in the sense you can make very granular changes that dont affect the entire “wifi network”.  It would mean a heck of a lot of SSID’s though and maybe a lot of policies… and im having enough trouble with a tiny update… LOL. 

 

You know, i think i will have it hidden.  We basically broadcast three SSID’s, and each one of those SSID’s are on a different VLAN.  This new policy im doing, im going to leave that on default, which is the default VLAN (VLAN1).  I know that devices wouldnt be able to connect, but why even present that to them.  

When i go into Additional Settings »>> Option Settings i guess its okay to stick with all the defaults it pops up for things like, “rates, filters, user profiles, voice WWM”, etc. and only check “Hide SSID (stealth mode).

 

Ill keep you posted, and thanks again,

Jason.

Userlevel 3

Hmmmm.. Creating the Policy and im in the Device Template “tab”.  The Spare AP is a AP650(AH) that im going to use.

 

Its asking me for a template name, which i filled in, but just below that field it shows “AP650 Template - please select at least one port…..”, here:

Its going to create a new template right? and not overwrite the existing AP650 template i use on all the other AP650(AH)’s, do you know?

 

Thanks,

J

Userlevel 4

Yes, in theory you could! Slight tangent to the topic, but instead of creating separate policies, if you wanted to have different APs broadcasting different SSIDs you’d just use the “assign ssids using classification rules” box in the wireless networks section. Creating a separate policy for every SSID would take a lot longer and a lot messier :)
Yeah, for the purposes of this we don’t need to connect to the SSID, the key thing is working out whether the new policy can be pushed to the AP, then adding bonjour to check it’s definitely that causing it to break.
Do you have a screenshot of your bonjour settings? Wondering if I can try and replicate it somehow...

Userlevel 4

Unless you’re using a custom port type that’s not uplink and Native VLAN1 (default) then I wouldn’t worry about the device template. 

Userlevel 3

Unless you’re using a custom port type that’s not uplink and Native VLAN1 (default) then I wouldn’t worry about the device template. 

Good.   Ill go back in, and hit cancel to clear out all the settings, etc. in that template tab. 

The AP’s are on our backbone vlan1 via Ethernet, but based on what SSID a device connects to they will be on a different vlan.  So, if you connect to SSID “A” then you are on vlan 5 and get an IP address that corresponds to that vlan.  You connect to SSID “B” then you are on vlan 10, and same thing, you get an IP address corresponding to that vlan.

 

oh, and here is a screenshot of the BJ GW settings within our default policy:

 

We basically have to offer that crappy Bonjour protocol all over the place, and I'm sure you know what a bugger that is to do, especially when you have VLANs.  I remember Apple saying they were working on some new type of Bonjour, where it could traverse VLANS, etc. That article died a horrible death about 8+ years ago… probably when they realized its just a crap, chatting, free POS to begin with, and not worth the investment to create one that would actually work and not bog down a network.

I know we could have used the dashes between a lot of those VLANS, but we had a heck of a time getting it to work, and it turned out to be a problem with our DHCP server, which we had to rebuild from scratch (heck of a lot of work… no fun at all).  Then the BG GW started working. 

Reply