Data Center (VDX)

Expand all | Collapse all

vdx 6740 firmware upgrade fails - ISSU is not supported to the target firmware version

Yulia Abitbul

Yulia Abitbul01-30-2019 14:33

Yulia Abitbul

Yulia Abitbul01-30-2019 15:02

Yulia Abitbul

Yulia Abitbul01-30-2019 16:42

  • 1.  vdx 6740 firmware upgrade fails - ISSU is not supported to the target firmware version

    Posted 01-30-2019 11:31
    hi gents,

    I have two vdx in logical chassis. I'm trying to upgrade 7.2.0a1 to nos7.3.0aa but process fails.
    Is this upgrade path not supported or I'm having a problem?
    I'm hoping non-disruptive upgrade is possible with ISSU or some other way.

    code:
    bottom# firmware download logical-chassis ftp host 10.214.234.2 user anonymous directory / rbridge-id 1-2
    Password: *********

    Following is the result of the sanity check on the specified nodes.

    Rbridge-id Sanity Result Current Version
    --------------------------------------------------------------
    1 Failed * 7.2.0a1
    2 Failed * 7.2.0a1

    * Details:
    Rbridge-id 1:

    ISSU is not supported to the target firmware version. Please specify coldboot option in the command-line for download.

    The preinstall script failed.

    Rbridge-id 2:

    ISSU is not supported to the target firmware version. Please specify coldboot option in the command-line for download.

    The preinstall script failed.

    Firmware download failed. Please address the failures and then initiate firmware download again.


  • 2.  RE: vdx 6740 firmware upgrade fails - ISSU is not supported to the target firmware version

    Posted 01-30-2019 11:38
    Hi,

    Upgrade from NOS 7.2x to 7.3x is a major upgrade and as such ISSU is not supported

    The only option is coldboot, in your example as below

    code:
    firmware download logical-chassis ftp host 10.214.234.2 user anonymous directory / rbridge-id 1-2 coldboot


    I would suggest that you upgrade each switch individually to minimize disruption to service

    code:
    firmware download logical-chassis ftp host 10.214.234.2 user anonymous directory / rbridge-id 2 coldboot


    Start with the non-principal switch and once this is upgraded then upgrade the principal


  • 3.  RE: vdx 6740 firmware upgrade fails - ISSU is not supported to the target firmware version

    Posted 01-30-2019 12:33
    I followed the advice, upgraded only one switch and I now see:

    code:
    2019/01/30-12:21:03, [VCS-1006], 3978, SW/0 | Active, ERROR, VDX6740, Event: VCS node rejoin, Coordinator IP: 10.214.234.95, VCS ID: 1,
    Status: rBridge ID 2 (10.214.234.91) failed to rejoin VCS cluster, Reason: There is schema mismatch.
    2019/01/30-12:21:53, [VCS-1006], 3979, SW/0 | Active, ERROR, VDX6740, Event: VCS node rejoin, Coordinator IP: 10.214.234.95, VCS ID: 1,
    Status: rBridge ID 2 (10.214.234.91) failed to rejoin VCS cluster, Reason: There is schema mismatch.


    Is it save to now upgrade second switch?


  • 4.  RE: vdx 6740 firmware upgrade fails - ISSU is not supported to the target firmware version

    Posted 01-30-2019 12:54
    Yes, schema mismatch means that switches are on different version codes and RB2 is waiting for RB1 to be upgraded


  • 5.  RE: vdx 6740 firmware upgrade fails - ISSU is not supported to the target firmware version

    Posted 01-30-2019 14:02
    This does not look good. It looks like "stack" lost configuration, lost Vlans, probably everything.
    How to troubleshoot this and how to fix!?

    many thanks.


  • 6.  RE: vdx 6740 firmware upgrade fails - ISSU is not supported to the target firmware version

    Posted 01-30-2019 14:33
    what "show vcs" tells you? from RB2?


  • 7.  RE: vdx 6740 firmware upgrade fails - ISSU is not supported to the target firmware version

    Posted 01-30-2019 14:56
    Both switches show online status:

    Rbridge-Id WWN Management IP VCS Status Fabric Status HostName
    --------------------------------------------------------------------------------------------------------------
    1 >10:00:C4:F5:7C:93 10.214.234.95 Online Online sw0
    2 10:00:C4:F5:7C:93* 10.214.234.91 Online Online sw0


  • 8.  RE: vdx 6740 firmware upgrade fails - ISSU is not supported to the target firmware version

    Posted 01-30-2019 14:56
    Slot Name Primary/Secondary Versions Status
    ---------------------------------------------------------------------------
    SW/0 NOS 7.3.0aa ACTIVE*
    7.3.0aa
    SW/1 NOS 7.3.0aa STANDBY
    7.3.0aa


  • 9.  RE: vdx 6740 firmware upgrade fails - ISSU is not supported to the target firmware version

    Posted 01-30-2019 15:02
    Both of them upgraded? But you lost config?


  • 10.  RE: vdx 6740 firmware upgrade fails - ISSU is not supported to the target firmware version

    Posted 01-30-2019 15:13
    Please re-apply configuration from back up
    To understand why cluster lost configuration after upgrade you need to collect Support files from both switches and open case to Extreme GTAC.

    Did you see any errors while upgrade?


  • 11.  RE: vdx 6740 firmware upgrade fails - ISSU is not supported to the target firmware version

    Posted 01-30-2019 16:25
    I was not watching.
    How do I collect most info that can help investigation?


  • 12.  RE: vdx 6740 firmware upgrade fails - ISSU is not supported to the target firmware version

    Posted 01-30-2019 16:42
    code:
    #copy support


    will have everything


  • 13.  RE: vdx 6740 firmware upgrade fails - ISSU is not supported to the target firmware version

    Posted 01-30-2019 19:58
    But in the mean while I'd like to say that it's quite a disaster the way the upgrade does not work.
    I had a pretty "regular" config, only ethernet, handful of vlans, nothing complex. I first upgraded the stand-by rb2 switch and then the active switch.

    I'd question validation, testing, the whole process frankly, which declares whether specific upgrade path is safe & ready.
    I can not understand how this was possible that upgrade lost configuration.


  • 14.  RE: vdx 6740 firmware upgrade fails - ISSU is not supported to the target firmware version

    Posted 01-30-2019 20:04
    You are right, you should not lose configuration after upgrade. Only if you used "default-config" option. Something went wrong
    That might be a Defect, that is why I suggested to collect Support files and open a ticket to GTAC


  • 15.  RE: vdx 6740 firmware upgrade fails - ISSU is not supported to the target firmware version

    Posted 01-31-2019 20:35
    Hi Pawel,

    NOS 7.3.0aa is a special release intended for FIP (Fed Customer), it is not a GA release for general public.
    The release note of NOS 7.3.0aa stated that it will not support ISSU upgrade, it requires default config.
    So the loss of config when you upgrade from 7.2.0a1 is expected.
    In additions, please do not upgrade to NOS 7.3.0aa, for general deployment NOS 7.3.0a is the latest and should not venture to NOS 7.3.0aa until further notice.
    Expect some doc modification on the portal to be made soon, that will make this point more easy for users to follow.


  • 16.  RE: vdx 6740 firmware upgrade fails - ISSU is not supported to the target firmware version

    Posted 02-01-2019 16:20
    thanks Ivan,

    you say "please do not upgrade to NOS 7.3.0aa" and I already had said I had done already - what do I do now?
    Downgrade to NOS 7.3.0a? Wait for new version?

    many thanks.


  • 17.  RE: vdx 6740 firmware upgrade fails - ISSU is not supported to the target firmware version

    Posted 02-01-2019 18:05
    Hi Pawel,

    Yes, please downgrade back to NOX 7.3.0a.
    NOS 7.3.0aa is NOT for general consumption.
    NOS 7.3.0a is the latest for GA ( while NOS 7.3.0aa is for specific customers with FIP requirement ), it will not do you any good running NOS 7.3.0aa

    The official word from GTAC and Engineering is that NOS 7.3.0a should be the one for general consumption, No one should recommend customer to upgrade to NOS 7.3.0aa.

    We have also communicated to other users who upgraded to NOS 7.3.0aa, and also ran into issue, to downgrade to NOS 7.3.0.


  • 18.  RE: vdx 6740 firmware upgrade fails - ISSU is not supported to the target firmware version

    Posted 02-01-2019 18:36
    Okey then - should I expect configuration to get cleared with the downgrade?

    My remark - it did not require "default config", upgrade process did not object. It would have been great if did but it proceeded with "cold reboot" as normal. Maybe such a safety-feature-check should be introduced into these "special" releases, so those of us who do not read release-notes would be saved from stumbling unexpectedly into a rabbit hole.


  • 19.  RE: vdx 6740 firmware upgrade fails - ISSU is not supported to the target firmware version

    Posted 02-01-2019 19:45
    I hear you.
    That was one of the issues. ISSU upgrade for 730aa is not supported but not clearly stated in Release_note. But upgrade from 7.2.0 to 7.3.0aa is stated that it requires default config. Furthermore, coldboot was accept and proceed to fwdl, but nuke the config ( you will see "DCM: database conversion failed" in console message that scrolled across the screen ); and if coldboot option is not used, the fwdl may leave the switch partitions in limbo, they are out-of-sync, requiring "firmware sync" to rescue.

    Folks are working on plugging those rabbit holes now and will make the necessary changes on the portal.

    You may want to save the config, before downgrade. I did not get a chance to check donwgrade from 730aa to 730a yet. In case you need to copy/paste back once it is fwdl to 730a.

    Thanks and sorry about the hiccup ...