SummitStack upgrade fails : "No space left on device"

  • 0
  • 1
  • Problem
  • Updated 1 year ago
  • Solved
Hi everybody,

I'm upgrading my production SummitStacks from EXOS version 15.1.4.3 to 16.1.3.6.
The upgrade was successful on every stack except one, which fails and displays this error message :

Failed to download image on Slot-7 - tftp: can't open '/scratch/summitX-16.1.3.6-patch1-11.xos': No space left on device

It's a 8-member stack, i tried to launch the upgrade several times and it always fails on Slot 7.

Here is the complete output displayed in CLI :



Slot-1  # download image X.X.X.X extreme/summitX-16.1.3.6-patch1-11.xos vr vr-Default secondary
Do you want to install image after downloading? (y - yes, n - no, <cr> - cancel) Yes
Downloading to Slot-1(Master): ..................................................................................................................................................................................
Downloading to Slot-6(Standby):
Downloading to Slot-5(Standby):
Downloading to Slot-7(Standby):
Downloading to Slot-8(Standby):
Downloading to Slot-2(Backup):
Downloading to Slot-3(Standby):
Downloading to Slot-4(Standby): Failed to download image on Slot-7 - tftp: can't open '/scratch/summitX-16.1.3.6-patch1-11.xos': No space left on device
.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*..***Error: Download image failed (see above error message)

I rebooted the slot 7 but it doesn't solve the problem.
I even tried to remove slot 7 from the stack, do "unconfigure switch all" on slot 7 to put it back to factory default, and then put it back in the stack. The issue is still here.


I wonder if any of you ever faced this problem.
Thanks in advance for the help

Gabriel
Photo of Gabriel

Gabriel

  • 410 Points 250 badge 2x thumb

Posted 1 year ago

  • 0
  • 1
Photo of Dave Hammers

Dave Hammers, Dir SW Engineering

  • 3,502 Points 3k badge 2x thumb
You could try BootRom rescue using the summitX-16.1.3.6-patch1-11.xos.
https://gtacknowledge.extremenetworks.com/articles/How_To/How-to-Use-the-Boot-Rom-Menu-to-Download-a...
Photo of Brad Parker

Brad Parker, Technical Support Engineer

  • 3,420 Points 3k badge 2x thumb
BootROM rescue, as Dave mentions, will wipe all of the memory on the switch as a part of the process. This is also a valid solution.
Photo of Brad Parker

Brad Parker, Technical Support Engineer

  • 3,420 Points 3k badge 2x thumb
Hi Gabriel,

Please first try to run a "ls" and "ls internal-memory" on the switch. Both of these will still retain files after "unconfigure switch all"

Slot-1 Stack.11 # ls internal-memory
-rwxr-xr-x    1 root     0           23427 Feb 25 09:59 show_tech.log.gz
-rwxr-xr-x    1 root     0          153482 Feb 25 09:58 trace.devmgr.1417
-rwxr-xr-x    1 root     0          135079 Feb 25 09:58 trace.nodemgr.1421
-rwxr-xr-x    1 root     0            5622 Feb 25 09:58 trace.vlan.1437

Then, at the prompt, type "rm internal-memory *.*"
Slot-1 Stack.17 # rm internal-memory *.*
NOTE: This will remove all files from internal storage.
Continue? (y/N) Yes
Slot-1 Stack.18 # -rwxr-xr-x    1 root     0          135079 Feb 25 09:58 trace.Slot-1 Stack.18 # -rwxr-xr-x    1 root     0          153482 Feb 25 09:58 trace.Slot-1 Stack.18 # -rwxr-xr-x    1 root     0           23427 Feb 25 09:59 show_tSlot-1 Stack.18 # ls internal-memory
 1K-blocks      Used Available Use%
     49038       716     48322   1%
Then, at the prompt, "ls" and "rm" to any files you feel comfortable removing. I wouldn't remove the configuration file that you're using.

Hope this helps.

Thanks
Brad
Photo of Gabriel

Gabriel

  • 410 Points 250 badge 2x thumb
Hello,

Thanks for your help.
When i run "ls" and "ls internal-memory" on the Master slot, it seems no files are using a lot of space.
Then from slot 1 I telnet into slot 7 and run the same commands, i find that internal memory has only 38% space left. I try to erase the content of internal memory but it seems I can't do that from a Standby slot :

Slot-7  > ls internal-memory
-rwxr-xr-x    1 root     0           35645 Feb 24 17:28 old_traces.tgz

 1K-blocks      Used Available Use%
    118160     44474     73686  38%
Slot-7  >
Slot-7  > rm internal-memory old_traces.tgz
Remove old_traces.tgz from internal storage? (y/N) Yes
This command cannot be executed on a standby node.

I ran the "rm internal-memory old_traces.tgz" on the Master slot but it didn't remove the files from slot 7.
Is there a way to remove files that are on a standby slot ?

Best regards,
Gabriel
Photo of Grosjean, Stephane

Grosjean, Stephane, Employee

  • 12,860 Points 10k badge 2x thumb
out of curiosity, what switch it is (slot 7)?
Photo of Gabriel

Gabriel

  • 410 Points 250 badge 2x thumb
Hi,

All slots are X460-48p

Gabriel
Photo of David Coglianese

David Coglianese, Embassador

  • 6,294 Points 5k badge 2x thumb
You could enable master-capability on slot 7 then unplug the stack cables. The switch should reboot as its own master and you could they run the rm commands again.

After the files are wiped disable master-capability on the 7 reboot it and and plug the stack cables back in.

Hope this helps,
Photo of Gabriel

Gabriel

  • 410 Points 250 badge 2x thumb
Hi David,

I'll try this, and tell you the result.

Thanks for the help

Gabriel
Photo of Drew C.

Drew C., Community Manager

  • 38,612 Points 20k badge 2x thumb
Hi Gabriel, were you able to free up some space and get EXOS to install on that slot?
Photo of Gabriel

Gabriel

  • 410 Points 250 badge 2x thumb
Hello,

I managed to remove the slot from the stack, remove files in internal-memory, and apply the new EXOS image.

Thanks everybody for your help

Gabriel