Switch temporary shown "offline" in XIQ, Firmwareupdate not possible via XIQ and USB+CLI

[TheManneken]

Servus and hello,

We have a lot of X435-8P-4S each at different locations, almost all 32.6.3.127 running. Some of the switches randomly go "offline" in XIQ and come back online after a few hours, while they are supplying network traffic properly and I couldn't get a reason for that, neither the logs or the firewall policy. That led me to upgrade the firmware to 32.7.1.9 , but even switches marked as online the Firmware Updating progress halts ab 8% and fails after 10-15 minutes.

So I went to one of the installation locations and tried to update via USB and file summitlite_arm-32.7.1.9-patch1-26.xos, but it fails too.

Copy 'summitlite_arm-32.7.1.9-patch1-26.xos' from '/usr/local/ext' to '/usr/local/tmp'? (y/N) Yes
cp: write error: No space left on device

Since the switch still had the older 31.7.1.4 patch1-98 in the other partition, I rebooted the switch to it and tried again. The switch even was shown online in XIQ after that, but went "offline" after like an hour

The events of switch now running 31.7.1.4 patch1-98 are (via SSH)

IQagent Upgrade : CLI:
Error: Failed to download image - No space left on device

Image upgrade operation failed : CLI: Note: The inactive partition (secondary) will be used for installation.
This image will be used only after rebooting the switch!
Error: Failed to install image - tar: write error: No space left on device

IQagent Upgrade : HiveAgent Upgrade Failure: timeout

And additionaly, I have now repeating new errors via Console immediately after boot up:

Spoiler

*** EXOS Boot Menu ( 2.0.0.6 ) ***

EXOS: Default
EXOS: Primary 31.7.1.4
EXOS: Secondary 32.7.1.9
EXOS: Primary 31.7.1.4 with default configuration
EXOS: Secondary 32.7.1.9 with default configuration
EXOS: Rescue
Run Operational Diagnostics
Run Manufacturing Diagnostics
Update bootloader
Reboot system

Press UP/DOWN to move, ENTER to select
Read 49152 bytes from volume bootnext to 64000000
## Info: input data size = 15 = 0xF
Read 49152 bytes from volume bootargs to 64000000
## Info: input data size = 344 = 0x158
Read 49152 bytes from volume bootscript to 64000000
## Info: input data size = 2656 = 0xA60
Set watchdog to 80 seconds
Read 0 bytes from volume kern1 to 64000000
No size specified -> Using max size (10597380)
## Loading kernel from FIT Image at 64000000 ...
Using 'X435-8P-4S' configuration
Trying 'kernel_iproc' kernel subimage
Description: Broadcom iProc Linux
Type: Kernel Image
Compression: uncompressed
Data Start: 0x640000d8
Data Size: 2520936 Bytes = 2.4 MiB
Architecture: ARM
OS: Linux
Load Address: 0x61008000
Entry Point: 0x61008000
Hash algo: crc32
Hash value: 5e4d1b79
Verifying Hash Integrity ... crc32+ OK
## Loading ramdisk from FIT Image at 64000000 ...
Using 'X435-8P-4S' configuration
Trying 'ramdisk-exos' ramdisk subimage
Description: EXOS init ramdisk
Type: RAMDisk Image
Compression: uncompressed
Data Start: 0x64267924
Data Size: 8049231 Bytes = 7.7 MiB
Architecture: ARM
OS: Linux
Load Address: 0x62000000
Entry Point: 0x002678c0
Hash algo: crc32
Hash value: 6e38ddd5
Verifying Hash Integrity ... crc32+ OK
Loading ramdisk from 0x64267924 to 0x62000000
## Loading fdt from FIT Image at 64000000 ...
Using 'X435-8P-4S' configuration
Trying 'fdt_X435_8TP' fdt subimage
Description: Extreme X435-8T/P-4S Device Tree blob - extreme_X435-8TP.dtb
Type: Flat Device Tree
Compression: uncompressed
Data Start: 0x64a16c8c
Data Size: 7926 Bytes = 7.7 KiB
Architecture: ARM
Hash algo: crc32
Hash value: b92732f5
Verifying Hash Integrity ... crc32+ OK
Booting using the fdt blob at 0x64a16c8c
Loading Kernel Image ... OK
Loading Ramdisk to 7f325000, end 7fad224f ... OK
Using Device Tree in place at 64a16c8c, end 64a1bb81

Starting kernel ...

Disabling outer cache

/
udevd: Starting version 244.5+
init started: BusyBox v1.31.1 ()
Starting ExtremeXOS 31.7.1b4
Copyright (C) 1996-2023 Extreme Networks, Inc. All rights reserved.
This product is protected by one or more US patents listed at https://www.extremenetworks.com/company/legal/patents/ along with their foreign counterparts.

(pending-AAA) login:

Authentication Service (AAA) on the master node is now available for login.

touch: /scratch//intern/dhcp/dhclient.leases: No space left on device
Traceback (most recent call last):
File "/usr/lib/python2.7/logging/__init__.py", line 892, in emit
File "/usr/lib/python2.7/logging/__init__.py", line 852, in flush
IOError: [Errno 28] No space left on device
Logged from file __main__.py, line 73
Traceback (most recent call last):
File "/usr/lib/python2.7/logging/__init__.py", line 892, in emit
File "/usr/lib/python2.7/logging/__init__.py", line 852, in flush
IOError: [Errno 28] No space left on device
Logged from file __main__.py, line 75

[SvenS]

You need to free up some space on the device.

Check if an diagnostics.xmod image is available:

show ver images
Card Partition Installation Date Version Name Branch
------------------------------------------------------------------------------
Switch primary Thu Jan 1 00:09:55 UTC 1970 31.7.1.4 rescue.xos 31.7.1.4-patch1-98
Switch primary Mon Aug 7 16:51:08 UTC 2023 31.7.1.4 summitlite_arm-31.7.1.4-patch1-98-diagnostics.xmod 31.7.1.4-patch1-98
Switch secondary Thu Apr 4 09:57:29 UTC 2024 32.6.3.127 summitlite_arm-32.6.3.127.xos 32.6.3.127

you can uninstall this image

uninstall image summitlite_arm-31.7.1.4-patch1-98-diagnostics.xmod primary

Now ist possible to upgrade the device manually via cli. 

[TheManneken]

The diagnostics image was present and already uninstalled before. There seems to be still not enough space available.

[Stefan_K_]

Can you provide the output of "ls /usr/local/tmp"?

Can you try these two different ways to install the image, instead of copying the image from the USB first?

1. install directly from the USB: 

install image /usr/local/ext/<filename>.xos

2. using download image command: 

download url file:///usr/local/ext/<filename>.xos

Best regards
Stefan

[TheManneken]

I managed to get the output:

* X435-8P-4S.1 # ls /usr/local/tmp
drwx------    2 root     root           224 Mar  5  2024 bcmsdk
drwxrwxrwx    2 root     root           376 Mar  7  2024 cc_logs
drwxrwxrwx    2 root     root           232 Mar  5  2024 dhcp
drwxr-xr-x    2 root     root           160 Jan  1  1970 diagnostics
drwxrwxrwx    2 root     root          1936 Sep 18 08:08 eciq
d-w-------    2 root     root           224 Jul 22 05:54 healthcheck
drwxrwxrwx    2 root     root           232 Sep 13 08:11 inlets
drwxrwxrwx    2 root     root           232 Aug  8 21:43 telegraf

 1K-blocks      Used Available Use%
     19324     19268        56 100%
* X435-8P-4S.2 #

* X435-8P-4S.5 # show switch mounts
Memory storage is present.
* X435-8P-4S.6 # ls /usr/local/ext
drwxrwxr--    2 admin    admin        16384 Sep 13 09:55 System Volume Information
-rwxrwxr--    1 admin    admin     51619755 Sep 13 10:06 summitlite_arm-32.7.1.9-patch1-26.xos

 1K-blocks      Used Available Use%
  30014464     50480  29963984   0%
* X435-8P-4S.7 # ls /usr/local/ext/
drwxrwxr--    2 admin    admin        16384 Sep 13 09:55 System Volume Information
-rwxrwxr--    1 admin    admin     51619755 Sep 13 10:06 summitlite_arm-32.7.1.9-patch1-26.xos

 1K-blocks      Used Available Use%
  30014464     50480  29963984   0%
* X435-8P-4S.8 # install image /usr/local/ext/summitlite_arm-32.7.1.9-patch1-26.xos
Note: The inactive partition (primary) will be used for installation.

Downloading to Switch..........................................................................................
Error: Failed to download image - No space left on device

* X435-8P-4S.9 # show ver images
Card  Partition     Installation Date        Version     Name      Branch
------------------------------------------------------------------------------
Switch primary   Mon Jul 8 19:05:37 UTC 2024 32.7.1.9 summitlite_arm-32.7.1.9.xos 32.7.1.9
Switch secondary Tue Apr 23 16:27:35 UTC 2024 32.7.1.9 summitlite_arm-32.7.1.9.xos 31.7.1.4-patch1-98

* X435-8P-4S.10 # download url file:///usr/local/ext/summitlite_arm-32.7.1.9.xos
Note: The inactive partition (primary) will be used for installation.
Do you want to install image after downloading? (y - yes, n - no, <cr> - cancel) Yes

Downloading to Switch
Error: Failed to download image - No space left on device

* X435-8P-4S.11 #