topic RE: vdx 6740 firmware upgrade fails - ISSU is not supported to the target firmware version in Data Center (VDX)

vdx 6740 firmware upgrade fails - ISSU is not supported to the target firmware version

Pawel_Eljasz — Wed, 30 Jan 2019 19:31:00 GMT

hi gents,

I have two vdx in logical chassis. I'm trying to upgrade 7.2.0a1 to nos7.3.0aa but process fails.
Is this upgrade path not supported or I'm having a problem?
I'm hoping non-disruptive upgrade is possible with ISSU or some other way.

code:

bottom# firmware download logical-chassis ftp host 10.214.234.2 user anonymous directory / rbridge-id 1-2
Password: *********

Following is the result of the sanity check on the specified nodes.

Rbridge-id    Sanity Result           Current Version               
--------------------------------------------------------------
1             Failed *                7.2.0a1
2             Failed *                7.2.0a1

* Details:
Rbridge-id 1:

ISSU is not supported to the target firmware version. Please specify coldboot option in the command-line for download.

The preinstall script failed.

Rbridge-id 2:

ISSU is not supported to the target firmware version. Please specify coldboot option in the command-line for download.

The preinstall script failed.

Firmware download failed. Please address the failures and then initiate firmware download again.

RE: vdx 6740 firmware upgrade fails - ISSU is not supported to the target firmware version

Mick_Day — Wed, 30 Jan 2019 19:38:00 GMT

Hi,

Upgrade from NOS 7.2x to 7.3x is a major upgrade and as such ISSU is not supported

The only option is coldboot, in your example as below

code:

firmware download logical-chassis ftp host 10.214.234.2 user anonymous directory / rbridge-id 1-2 coldboot

I would suggest that you upgrade each switch individually to minimize disruption to service

code:

firmware download logical-chassis ftp host 10.214.234.2 user anonymous directory / rbridge-id 2 coldboot

Start with the non-principal switch and once this is upgraded then upgrade the principal

RE: vdx 6740 firmware upgrade fails - ISSU is not supported to the target firmware version

Pawel_Eljasz — Wed, 30 Jan 2019 20:33:00 GMT

I followed the advice, upgraded only one switch and I now see:

code:

2019/01/30-12:21:03, [VCS-1006], 3978, SW/0 | Active, ERROR, VDX6740, Event: VCS node rejoin, Coordinator IP: 10.214.234.95, VCS ID: 1,
 Status: rBridge ID 2 (10.214.234.91) failed to rejoin VCS cluster, Reason: There is schema mismatch.
2019/01/30-12:21:53, [VCS-1006], 3979, SW/0 | Active, ERROR, VDX6740, Event: VCS node rejoin, Coordinator IP: 10.214.234.95, VCS ID: 1,
 Status: rBridge ID 2 (10.214.234.91) failed to rejoin VCS cluster, Reason: There is schema mismatch.

Is it save to now upgrade second switch?

RE: vdx 6740 firmware upgrade fails - ISSU is not supported to the target firmware version

Yulia_Abitbul — Wed, 30 Jan 2019 20:54:00 GMT

Yes, schema mismatch means that switches are on different version codes and RB2 is waiting for RB1 to be upgraded

RE: vdx 6740 firmware upgrade fails - ISSU is not supported to the target firmware version

Pawel_Eljasz — Wed, 30 Jan 2019 22:02:00 GMT

This does not look good. It looks like "stack" lost configuration, lost Vlans, probably everything.
How to troubleshoot this and how to fix!?

many thanks.

RE: vdx 6740 firmware upgrade fails - ISSU is not supported to the target firmware version

Yulia_Abitbul — Wed, 30 Jan 2019 22:33:00 GMT

what "show vcs" tells you? from RB2?

RE: vdx 6740 firmware upgrade fails - ISSU is not supported to the target firmware version

Pawel_Eljasz — Wed, 30 Jan 2019 22:56:00 GMT

Both switches show online status:

Rbridge-Id WWN Management IP VCS Status Fabric Status HostName
--------------------------------------------------------------------------------------------------------------
1 >10:00:C4:F5:7C:93 10.214.234.95 Online Online sw0
2 10:00:C4:F5:7C:93* 10.214.234.91 Online Online sw0

RE: vdx 6740 firmware upgrade fails - ISSU is not supported to the target firmware version

Pawel_Eljasz — Wed, 30 Jan 2019 22:56:00 GMT

Slot Name Primary/Secondary Versions Status
---------------------------------------------------------------------------
SW/0 NOS 7.3.0aa ACTIVE*
7.3.0aa
SW/1 NOS 7.3.0aa STANDBY
7.3.0aa

RE: vdx 6740 firmware upgrade fails - ISSU is not supported to the target firmware version

Yulia_Abitbul — Wed, 30 Jan 2019 23:02:00 GMT

Both of them upgraded? But you lost config?

RE: vdx 6740 firmware upgrade fails - ISSU is not supported to the target firmware version

Yulia_Abitbul — Wed, 30 Jan 2019 23:13:00 GMT

Please re-apply configuration from back up
To understand why cluster lost configuration after upgrade you need to collect Support files from both switches and open case to Extreme GTAC.

Did you see any errors while upgrade?

RE: vdx 6740 firmware upgrade fails - ISSU is not supported to the target firmware version

Pawel_Eljasz — Thu, 31 Jan 2019 00:25:00 GMT

I was not watching.
How do I collect most info that can help investigation?

RE: vdx 6740 firmware upgrade fails - ISSU is not supported to the target firmware version

Yulia_Abitbul — Thu, 31 Jan 2019 00:42:00 GMT

code:

#copy support

will have everything

RE: vdx 6740 firmware upgrade fails - ISSU is not supported to the target firmware version

Pawel_Eljasz — Thu, 31 Jan 2019 03:58:00 GMT

But in the mean while I'd like to say that it's quite a disaster the way the upgrade does not work.
I had a pretty "regular" config, only ethernet, handful of vlans, nothing complex. I first upgraded the stand-by rb2 switch and then the active switch.

I'd question validation, testing, the whole process frankly, which declares whether specific upgrade path is safe & ready.
I can not understand how this was possible that upgrade lost configuration.

RE: vdx 6740 firmware upgrade fails - ISSU is not supported to the target firmware version

Yulia_Abitbul — Thu, 31 Jan 2019 04:04:00 GMT

You are right, you should not lose configuration after upgrade. Only if you used "default-config" option. Something went wrong
That might be a Defect, that is why I suggested to collect Support files and open a ticket to GTAC

RE: vdx 6740 firmware upgrade fails - ISSU is not supported to the target firmware version

Ivan_Chan — Fri, 01 Feb 2019 04:35:00 GMT

Hi Pawel,

NOS 7.3.0aa is a special release intended for FIP (Fed Customer), it is not a GA release for general public.
The release note of NOS 7.3.0aa stated that it will not support ISSU upgrade, it requires default config.
So the loss of config when you upgrade from 7.2.0a1 is expected.
In additions, please do not upgrade to NOS 7.3.0aa, for general deployment NOS 7.3.0a is the latest and should not venture to NOS 7.3.0aa until further notice.
Expect some doc modification on the portal to be made soon, that will make this point more easy for users to follow.

RE: vdx 6740 firmware upgrade fails - ISSU is not supported to the target firmware version

Pawel_Eljasz — Sat, 02 Feb 2019 00:20:00 GMT

thanks Ivan,

you say "please do not upgrade to NOS 7.3.0aa" and I already had said I had done already - what do I do now?
Downgrade to NOS 7.3.0a? Wait for new version?

many thanks.

RE: vdx 6740 firmware upgrade fails - ISSU is not supported to the target firmware version

Ivan_Chan — Sat, 02 Feb 2019 02:05:00 GMT

Hi Pawel,

Yes, please downgrade back to NOX 7.3.0a.
NOS 7.3.0aa is NOT for general consumption.
NOS 7.3.0a is the latest for GA ( while NOS 7.3.0aa is for specific customers with FIP requirement ), it will not do you any good running NOS 7.3.0aa

The official word from GTAC and Engineering is that NOS 7.3.0a should be the one for general consumption, No one should recommend customer to upgrade to NOS 7.3.0aa.

We have also communicated to other users who upgraded to NOS 7.3.0aa, and also ran into issue, to downgrade to NOS 7.3.0.

RE: vdx 6740 firmware upgrade fails - ISSU is not supported to the target firmware version

Pawel_Eljasz — Sat, 02 Feb 2019 02:36:00 GMT

Okey then - should I expect configuration to get cleared with the downgrade?

My remark - it did not require "default config", upgrade process did not object. It would have been great if did but it proceeded with "cold reboot" as normal. Maybe such a safety-feature-check should be introduced into these "special" releases, so those of us who do not read release-notes would be saved from stumbling unexpectedly into a rabbit hole.

RE: vdx 6740 firmware upgrade fails - ISSU is not supported to the target firmware version

Ivan_Chan — Sat, 02 Feb 2019 03:45:00 GMT

I hear you.
That was one of the issues. ISSU upgrade for 730aa is not supported but not clearly stated in Release_note. But upgrade from 7.2.0 to 7.3.0aa is stated that it requires default config. Furthermore, coldboot was accept and proceed to fwdl, but nuke the config ( you will see "DCM: database conversion failed" in console message that scrolled across the screen ); and if coldboot option is not used, the fwdl may leave the switch partitions in limbo, they are out-of-sync, requiring "firmware sync" to rescue.

Folks are working on plugging those rabbit holes now and will make the necessary changes on the portal.

You may want to save the config, before downgrade. I did not get a chance to check donwgrade from 730aa to 730a yet. In case you need to copy/paste back once it is fwdl to 730a.

Thanks and sorry about the hiccup ...