topic Re: Problems with new AP230s refusing to take config? in Aerohive Migrated Content

Problems with new AP230s refusing to take config?

braden_mcgrath — Sat, 05 May 2018 03:42:37 GMT

I have had a rash of brand new AP230s that refuse to accept config out of the box. Additionally, they don't even seem to listen on port 22 (SSH) or 80/443 (web/https) for any sort of admin access, but they do come up enough to check-in via CAPWAP.

I have yet to get one physically in my hands to attach a console cable to it, they have all been direct-shipped to sites for 3rd-party contractors to install, and then after hanging them I discover that usually 1-2 APs per site (out of 10-15 total) pull this stunt and refuse to accept new config.

Oddly, I can push firmware to them without a problem, but no matter what I do config-wise, they won't accept anything.

Just wondering if anybody else in the community has seen this recently with AP230s... It's very frustrating and has greatly impacted the speed / smoothness of this rollout project. Never had issues like this with AP120 & AP121.

Re: Problems with new AP230s refusing to take config?

AnonymousM — Mon, 07 May 2018 12:57:04 GMT

As you have a CAPWAP connection with them, can you open an SSH tunnel and run a factory default reset ("reset config") (-> assuming that they are not pre-configured with special IP settings....).

Otherwise this sounds more like a Firewall issue (not fully allowing outgoing traffic on UDP 12222 and TCP 443, and - if Classic - TCP 22).

Re: Problems with new AP230s refusing to take config?

braden_mcgrath — Tue, 08 May 2018 05:20:21 GMT

Carsten - great idea, but as far as I can tell, this requires SSH to actually be listening on the AP itself, which it isn't. I can portscan all of the new APs at a site (it's not a firewall issue), and only the one or two that are "misbehaving" (refusing to take config) will show that SSH and HTTP/HTTPS are closed.

When I try to use Hivemanager (we have an on-prem virtual HM install) to SSH to a "bad" AP, I get the error: "Cannot set up an SSH connection between the device and HiveManager because the connection was refused."

It's like there is just something flat-out wrong with the APs. As I said previously, they will take a firmware update, but they still come back with this strange zombie unconfigurable state after rebooting from the update.

I'm waiting for a couple to be shipped back to me at HQ so I can get a console cable on them and try to see if they can be revived that way... but given that I've seen this with at least 6-8 AP230s now (across ~100-120 ordered in the past month or so), it's kind of a high failure / problem rate. Support has been willing to RMA them for me since they are non-functional from the factory, but that then means I have to schedule more remote hands teams to swap out the "DOA" units for the RMA replacements, since there is seemingly no way to revive these things remotely.

Re: Problems with new AP230s refusing to take config?

braden_mcgrath — Tue, 08 May 2018 05:24:24 GMT

When I try to push config (full upload, not just partial) from Hivemanager, the error message is:

"Upload Configuration: An error occurred when executing the following CLI command: config rollback next-reboot. Unknown error"

Incredibly helpful, no? 

Re: Problems with new AP230s refusing to take config?

AnonymousM — Tue, 08 May 2018 11:45:11 GMT

Braden,

Anything you do via the Hivemanager is a connection that is established from the AP, to the Hivemanager. Which is CAPWAP (UDP 12222 per default) for normal status reports and delta config uploads), and TCP 22 with HM Classic for SSH client, Firmware downloads and full config uploads. With NG this has changed to TCP 443. So in this case you do not need the AP to listen on 22, 80 or 443 - but I agree that it normally should. It's very odd though you say you can push a firmware, but not a full config or establish the ssh client...

Is your Hivemanager Classic or NG?

Ultimately you'd need console access, for sure. Until then you can try the following:

Ask someone onsite to push the reset button for 10-15 seconds (-> factory default reset)
"Reset device to default" via HM GUI
"Set the image to boot" via HM GUI, and choose the backup image. When it's back up and still not accessible, try the default reset again.
Push an older Firmware, try again.

Good luck 😕

Re: Problems with new AP230s refusing to take config?

braden_mcgrath — Tue, 08 May 2018 11:52:45 GMT

Carsten - we are on HM Classic. (I have no idea how we would even move to NG, since Aerohive's documentation on all of this is basically nonexistent.)

It was my understanding that some commands are sent via the CAPWAP connection, while others (like firmware download) were accomplished by the AP connecting back to the Hivemanager via SCP or TFTP (e.g. firmware download).

IME, a fresh out of box and correctly functioning AP will be listening on HTTP (and maybe HTTPS, can't remember) as well as SSH, so you can set one up without a Hivemanager if needed. The "bad" APs I've seen (refusing to accept config) are completely closed, port-wise. They respond to ping, and they talk to the HM via CAPWAP, but that's about it.

"Reset device to default" fails because the device would first need to have a config pushed to it - I get an error message along these lines from Hivemanager.

I've tried having someone do the factory-default with the reset button, and it doesn't seem to help.

Haven't tried "set image to boot" or pushing an older firmware, going to try those. These have generally been arriving with 6.5r7 or 6.5r8, IIRC. Pushing newer firmware will get it to reboot and check back in with new firmware running... but still won't accept config.

Re: Problems with new AP230s refusing to take config?

braden_mcgrath — Tue, 08 May 2018 11:54:46 GMT

As an example, I can't "diagnostics > show running config", because a full config has never been pushed to the unit (and Hivemanager comes back with an error along those lines).

Yeah, "set image to boot" doesn't work for the same reason:

"The requested operation cannot be performed until a complete configuration is uploaded to the device and it is rebooted to activate it."

Re: Problems with new AP230s refusing to take config?

braden_mcgrath — Tue, 08 May 2018 12:32:46 GMT

I mis-typed earlier... the problem APs do have HTTP & HTTPS listening, but when you try to access them with a browser, you get "ERR_EMPTY_RESPONSE" via HTTP, and "ERR_CONNECTION_CLOSED" via HTTPS.

The whole situation is really weird.

Re: Problems with new AP230s refusing to take config?

AnonymousM — Tue, 08 May 2018 14:40:00 GMT

Yeah, you really need to get access to one of them via console port, and then try all the previously mentioned steps again.

You mentioned earlier that Aerohive agreed to change all those APs, so I am assuming you have a support ticket open? If not, now would be a good point to do so 🙂

I vaguely remember having had a similar issue a long time ago, and it got fixed by a combination of pushing a different firmware, default resets, and pushing the final firmware version again.

Re: Problems with new AP230s refusing to take config?

braden_mcgrath — Tue, 15 May 2018 03:19:41 GMT

Carsten,

Figured you'd be interested in the follow-up here, I now finally have a couple of these problem APs in my hands and have serial access. 🙂

Here's the boot output... it seems like something is unhappy, storage-wise.

------------------------

U-Boot 2012.10 (Apr 13 2017 - 19:59:01)

I2C: ready

DEV ID= 0000cf12

SKU ID = 0

OTP status: eca00018

MEMC 0 DDR speed = 800MHz

Log: ddr40_phy_init.c: Configuring DDR Controller PLLs

Log: offset = 0x18010800

Log: VCO_FREQ is 1600 which is greater than 1Ghz.

Log: DDR Phy PLL polling for lock

Log: DDR Phy PLL locked.

Log: ddr40_phy_init::DDR PHY step size calibration complete.

Log: ddr40_phy_init:: Virtual VttSetup onm CONNECT=0x01CF7FFF, OVERRIDE=0x00077FFF

Log: ddr40_phy_init:: Virtual Vtt Enabled

Log: DDR Controller PLL Configuration Complete

PHY register dump after DDR PHY init

PHY register dump after mode register write

DRAM: 256 MiB

WARNING: Caches not enabled

GPIO Init ... Done

Power Input Detection: POE AF, Drive GPIO17(USB 5V enable) success

NAND: (ONFI), W29N04GV,

*ERROR* Invalid strap options for this NAND: page=1 type=2

Overriding invalid strap options: strap_type=2

128 KiB blocks, 2 KiB pages, 16B OOB, 8-bit

NAND: chipsize 512 MiB

Using default environment

In: serial

Out: serial

Err: serial

Unlocking L2 Cache ...Done

arm_clk=1000MHz, axi_clk=500MHz, apb_clk=250MHz, arm_periph_clk=500MHz

Net: Registering eth

Broadcom BCM IPROC Ethernet driver 0.1

Using GMAC1 (0x18025000)

et0: ethHw_chipAttach: Chip ID: 0xcf12; phyaddr: 0x1e

bcm_robo_attach: devid: 0x53012

bcmiproc_eth-0

MAC address is c867:5e6c:6140

Reset TPM chip...

Hit any key to stop autoboot: 0

NAND read: device 0 offset 0xa80000, size 0x500000

5242880 bytes read: OK

NAND read: device 0 offset 0x3780000, size 0x2800000

41943040 bytes read: OK

## Booting kernel from Legacy Image at 01005000 ...

Image Name: Linux-2.6.36

Image Type: ARM Linux Kernel Image (uncompressed)

Data Size: 2067124 Bytes = 2 MiB

Load Address: 80008000

Entry Point: 80008000

Verifying Checksum ... OK

## Loading init Ramdisk from Legacy Image at 02005000 ...

Image Name: uboot initramfs rootfs

Image Type: ARM Linux RAMDisk Image (uncompressed)

Data Size: 25899008 Bytes = 24.7 MiB

Load Address: 00000000

Entry Point: 00000000

Verifying Checksum ... OK

Power off two PHY...

Loading Kernel Image ... OK

boot_prep_linux commandline: root=/dev/ram console=ttyS0,9600 ramdisk_size=70000 cache-sram-size=0x10000

Using machid 0xbb8 from environment

Starting kernel ...

Uncompressing Linux... done, booting the kernel.

Mounting local file systems...

mount: mounting /dev/mtdblock8 on /f failed: Input/output error

mkdir: can't create directory '/f/system_info': Read-only file system

cp: can't create '/f/system_info/sw_ver': No such file or directory

open(/f/system_info/hw_info) failed.: No such file or directory

eth0: Broadcom BCM47XX 10/100/1000 Mbps Ethernet Controller 6.37.14.54 (r425351)

eth1: Broadcom BCM47XX 10/100/1000 Mbps Ethernet Controller 6.37.14.54 (r425351)

BCM radio init config

BCM radio init config done

Aerohive Wireless Extension 2.0.0 ... INIT!

Aerohive Forwarding Engine module ... INIT!

Image L7 repository version 5, checking existing L7 repository... ...

Updating L7 repository to version 5

mkdir: can't create directory '/f/l7': Read-only file system

mkdir: can't create directory '/f/l7/plugins': No such file or directory

cp: can't create '/f/l7/plugins': No such file or directory

tar: chdir(/f/l7/plugins/): No such file or directory

L7 repository has been updated to version 5

Verifying L7 data integrity

/opt/ah/etc/ah_start: cd: line 509: can't cd to /f/l7/plugins

/opt/ah/etc/ah_start: line 509: can't create tmp_plg.md5: Read-only file system

/opt/ah/etc/ah_start: line 509: can't create md5_diff: Read-only file system

/opt/ah/etc/ah_start: line 509: can't open md5_diff: no such file

L7 data integrity verified

cp: can't create '/f/system_info/kallsyms': No such file or directory

PM: create directory /f/etc/ failed.: Read-only file system

fopen(/f/etc/fek) failed: No such file or directory(2): No such file or directory

Fail to initialize the FEK: error code = -1: No such file or directory

failed to open file /f/pm_file0

System ready.

mkdir: can't create directory '/f/etc': Read-only file system

failed to open file /f/pm_file0

Could not load host key: /etc/ssh_host_rsa_key

Could not l

Welcome to Aerohive Product

AH-6c6140 login: failed to open file /f/pm_file0

failed to open file /f/pm_file0

-----------------

Interestingly, I am able to login to it (although it keeps spitting out the "failed to open file /f/pm_file0" on the console every so often).

Once I login, I tried:

AH-6c6140#reset config

Do you really want to reset the configuration to the default settings or, if set,to the bootstrap configuration?(Y/N)y

ERROR: reset config failed!

So it seems like they are going to need RMA after all. 

Re: Problems with new AP230s refusing to take config?

AnonymousM — Tue, 15 May 2018 11:39:56 GMT

Yeah, definitely not happy....

Did you try to boot the backup image, with default config (after reboot)?

load config default

reboot backup

Worth a try... but I agree, if this is about more than 1 or 2 APs, then an RMA is your better choice.

Re: Problems with new AP230s refusing to take config?

AnonymousM — Tue, 15 May 2018 11:41:10 GMT

I forgot to add: In case you are looking for the description of those CLI commands:

http://docs.aerohive.com/330000/docs/help/english/documentation/cli_guide_ap230_8-1r1.htm