topic RE: SummitStack upgrade fails : "No space left on device" in ExtremeSwitching (Other)

SummitStack upgrade fails : "No space left on device"

Gabriel3 — Sat, 25 Feb 2017 21:38:00 GMT

Hi everybody,

I'm upgrading my production SummitStacks from EXOS version 15.1.4.3 to 16.1.3.6.
The upgrade was successful on every stack except one, which fails and displays this error message :

Failed to download image on Slot-7 - tftp: can't open '/scratch/summitX-16.1.3.6-patch1-11.xos': No space left on device

It's a 8-member stack, i tried to launch the upgrade several times and it always fails on Slot 7.

Here is the complete output displayed in CLI :

Slot-1 # download image X.X.X.X extreme/summitX-16.1.3.6-patch1-11.xos vr vr-Default secondary
Do you want to install image after downloading? (y - yes, n - no, - cancel) Yes
Downloading to Slot-1(Master): ..................................................................................................................................................................................
Downloading to Slot-6(Standby):
Downloading to Slot-5(Standby):
Downloading to Slot-7(Standby):
Downloading to Slot-8(Standby):
Downloading to Slot-2(Backup):
Downloading to Slot-3(Standby):
Downloading to Slot-4(Standby): Failed to download image on Slot-7 - tftp: can't open '/scratch/summitX-16.1.3.6-patch1-11.xos': No space left on device
.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*..***Error: Download image failed (see above error message)

I rebooted the slot 7 but it doesn't solve the problem.
I even tried to remove slot 7 from the stack, do "unconfigure switch all" on slot 7 to put it back to factory default, and then put it back in the stack. The issue is still here.

I wonder if any of you ever faced this problem.
Thanks in advance for the help

Gabriel

RE: SummitStack upgrade fails : "No space left on device"

Dave_Hammers — Sat, 25 Feb 2017 22:01:00 GMT

You could try BootRom rescue using the summitX-16.1.3.6-patch1-11.xos.
https://gtacknowledge.extremenetworks.com/articles/How_To/How-to-Use-the-Boot-Rom-Menu-to-Download-a...

RE: SummitStack upgrade fails : "No space left on device"

BradP — Sat, 25 Feb 2017 22:01:00 GMT

BootROM rescue, as Dave mentions, will wipe all of the memory on the switch as a part of the process. This is also a valid solution.

RE: SummitStack upgrade fails : "No space left on device"

BradP — Sat, 25 Feb 2017 22:04:00 GMT

Hi Gabriel,

Please first try to run a "ls" and "ls internal-memory" on the switch. Both of these will still retain files after "unconfigure switch all"

Slot-1 Stack.11 # ls internal-memory
-rwxr-xr-x 1 root 0 23427 Feb 25 09:59 show_tech.log.gz
-rwxr-xr-x 1 root 0 153482 Feb 25 09:58 trace.devmgr.1417
-rwxr-xr-x 1 root 0 135079 Feb 25 09:58 trace.nodemgr.1421
-rwxr-xr-x 1 root 0 5622 Feb 25 09:58 trace.vlan.1437
Then, at the prompt, type "rm internal-memory *.*"
Slot-1 Stack.17 # rm internal-memory *.*
NOTE: This will remove all files from internal storage.
Continue? (y/N) Yes
Slot-1 Stack.18 # -rwxr-xr-x 1 root 0 135079 Feb 25 09:58 trace.Slot-1 Stack.18 # -rwxr-xr-x 1 root 0 153482 Feb 25 09:58 trace.Slot-1 Stack.18 # -rwxr-xr-x 1 root 0 23427 Feb 25 09:59 show_tSlot-1 Stack.18 # ls internal-memory
1K-blocks Used Available Use%
49038 716 48322 1% Then, at the prompt, "ls" and "rm" to any files you feel comfortable removing. I wouldn't remove the configuration file that you're using.

Hope this helps.

Thanks
Brad

RE: SummitStack upgrade fails : "No space left on device"

Gabriel3 — Sun, 26 Feb 2017 22:14:00 GMT

Hello,

Thanks for your help.
When i run "ls" and "ls internal-memory" on the Master slot, it seems no files are using a lot of space.
Then from slot 1 I telnet into slot 7 and run the same commands, i find that internal memory has only 38% space left. I try to erase the content of internal memory but it seems I can't do that from a Standby slot :

Slot-7 > ls internal-memory
-rwxr-xr-x 1 root 0 35645 Feb 24 17:28 old_traces.tgz

1K-blocks Used Available Use%
118160 44474 73686 38%
Slot-7 >
Slot-7 > rm internal-memory old_traces.tgz
Remove old_traces.tgz from internal storage? (y/N) Yes
This command cannot be executed on a standby node.

I ran the "rm internal-memory old_traces.tgz" on the Master slot but it didn't remove the files from slot 7.
Is there a way to remove files that are on a standby slot ?

Best regards,
Gabriel

RE: SummitStack upgrade fails : "No space left on device"

Stephane_Grosj1 — Mon, 27 Feb 2017 04:01:00 GMT

out of curiosity, what switch it is (slot 7)?

RE: SummitStack upgrade fails : "No space left on device"

Gabriel3 — Mon, 27 Feb 2017 17:05:00 GMT

Hi,

All slots are X460-48p

Gabriel

RE: SummitStack upgrade fails : "No space left on device"

davidj_cogliane — Mon, 27 Feb 2017 22:49:00 GMT

You could enable master-capability on slot 7 then unplug the stack cables. The switch should reboot as its own master and you could they run the rm commands again.

After the files are wiped disable master-capability on the 7 reboot it and and plug the stack cables back in.

Hope this helps,

RE: SummitStack upgrade fails : "No space left on device"

Gabriel3 — Tue, 28 Feb 2017 00:21:00 GMT

Hi David,

I'll try this, and tell you the result.

Thanks for the help

Gabriel

RE: SummitStack upgrade fails : "No space left on device"

Drew_C — Sat, 04 Mar 2017 03:58:00 GMT

Hi Gabriel, were you able to free up some space and get EXOS to install on that slot?

RE: SummitStack upgrade fails : "No space left on device"

Gabriel3 — Mon, 13 Mar 2017 14:50:00 GMT

Hello,

I managed to remove the slot from the stack, remove files in internal-memory, and apply the new EXOS image.

Thanks everybody for your help

Gabriel