topic Re: BR-MLX-MR2-M became an X card after firmware update in Data Center (MLX & CER/CES)

BR-MLX-MR2-M became an X card after firmware update

Aleksandr — Fri, 17 Dec 2021 13:11:33 GMT

Hello!

We had BR-MLX-MR2-M card with 5.6h firmware. (inside mlxe4)
We've put it in different chassis (mlxe16) as a secondary MGMT.

In this new chassis master MGMT has firmware v 6.2(b), after second card was synced it became a BR-MLX-MR2-X.
M1 (upper):BR-MLX-MR2-M Management Module Active
M2 (lower):BR-MLX-MR2-X Management Module Standby (Reset State)

And of course it gives us an error: Standby management module is not enabled due to type mismatch

Now question - what to do in such situation? As I understand the card just has an identity crisis.

Re: BR-MLX-MR2-M became an X card after firmware update

joergkost — Sat, 18 Dec 2021 12:34:35 GMT

Can you post the outputs of

show version management

show module

show running-config | include system

rconsole standby

There is also a known issue for placing management boards with older firmware versions to a chassis with newer firmware. Did you check this kb?

https://extremeportal.force.com/ExtrArticleDetail?an=000063714

Re: BR-MLX-MR2-M became an X card after firmware update

Aleksandr — Sat, 18 Dec 2021 12:49:30 GMT

show version management
System Mode: MLX
Chassis: MLXe 16-slot (Serial #: BFQ2546L00L, Part #: 40-1000360-05)
NI-X-HSF Switch Fabric Module 1 (Serial #: BEU0428L00E, Part #: 60-1001588-14)
FE 1: Type fe600, Version 1
FE 2: Type fe600, Version 1
FE 3: Type fe600, Version 1
Switch Fabric Module 1 Up Time is 2 days 13 hours 56 minutes 33 seconds
NI-X-HSF Switch Fabric Module 2 (Serial #: BEU0426L0BZ, Part #: 60-1001588-14)
FE 1: Type fe600, Version 1
FE 2: Type fe600, Version 1
FE 3: Type fe600, Version 1
Switch Fabric Module 2 Up Time is 2 days 13 hours 56 minutes 33 seconds
NI-X-HSF Switch Fabric Module 3 (Serial #: BEU0434L018, Part #: 60-1001588-14)
FE 1: Type fe600, Version 1
FE 2: Type fe600, Version 1
FE 3: Type fe600, Version 1
Switch Fabric Module 3 Up Time is 2 days 13 hours 56 minutes 33 seconds
NI-X-HSF Switch Fabric Module 4 (Serial #: BEU0316F08Z, Part #: 60-1001588-08)
FE 1: Type fe600, Version 1
FE 2: Type fe600, Version 1
FE 3: Type fe600, Version 1
Switch Fabric Module 4 Up Time is 2 days 13 hours 56 minutes 33 seconds
==========================================================================
SL M1: BR-MLX-MR2-M Management Module Active (Serial #: BVP0419L02Z, Part #: 60-1002374-07):
Boot     : Version 5.9.0T165 Copyright (c) 1996-2015 Brocade Communications Systems, Inc.
Compiled on Mar 19 2015 at 03:16:46 labeled as xmprm05900
(521771 bytes) from boot flash
Monitor : Version 6.2.0T165 Copyright (c) 1996-2015 Brocade Communications Systems, Inc.
Compiled on Aug 17 2017 at 11:22:12 labeled as xmb06200
(546965 bytes) from code flash
IronWare : Version 6.2.0bT163 Copyright (c) 1996-2015 Brocade Communications Systems, Inc.
Compiled on Apr 4 2018 at 08:59:36 labeled as xmr06200b
(10676645 bytes) from Primary
Board ID : 00 MBRIDGE Revision : 37
1666 MHz Power PC processor 7448 (version 8004/0202) 166 MHz bus
512 KB Boot Flash (MX29LV040C), 128 MB Code Flash (MT28F256J3)
4096 MB DRAM INSTALLED
4096 MB DRAM ADDRESSABLE
Active Management uptime is 2 days 13 hours 56 minutes 33 seconds

######################################################
show module
Module                                                          Status                       Ports    Starting MAC
M1 (upper):BR-MLX-MR2-M Management Module                       Active
M2 (lower):BR-MLX-MR2-X Management Module                       Standby(Reset State)
F1: NI-X-HSF Switch Fabric Module                               Active
F2: NI-X-HSF Switch Fabric Module                               Active
F3: NI-X-HSF Switch Fabric Module                               Active
F4: NI-X-HSF Switch Fabric Module                               Active
S1: NI-MLX-10Gx8-M 8-port 10GbE (M) Module                      CARD_STATE_UP                8        cc4e.2497.6900
S2: NI-MLX-10Gx8-M 8-port 10GbE (M) Module                      CARD_STATE_UP                8        cc4e.2497.6930
S3: NI-MLX-10Gx8-M 8-port 10GbE (M) Module                      CARD_STATE_UP                8        cc4e.2497.6960
S4: NI-MLX-10Gx8-M 8-port 10GbE (M) Module                      CARD_STATE_UP                8        cc4e.2497.6990
S5: Powered Off (By User)
S6:
S7:
S8:
S9: Powered Off (By User)
S10: Blocked for full height card in Slot 9
S11:
S12:
S13:
S14:
S15:
S16:
###############################################
show running-config | include system
system-max mac 131072
system-max vlan 1500
system-max ip-arp 65535
system-max ip-cache 256000
system-max ip-route 256000
system-max virtual-interface 1500
system-max ipv6-cache 65536
system-max ipv6-route 65536
system-max receive-cam 0
system-max ip-source-guard-cam 30000
system-max ipv6-receive-cam 4096
system-init tm-credit-size credit_1024b

#####################################################
Rconsole standby
Remote connection to standby established
Press CTRL-X to disconnect it
NetIron XMR/MLX Boot Monitor Version 6.2.0
Enter 'b' to stop at boot monitor
sent IPC_MSGTYPE_REBOOT to slot 16 (my_slot = 33, ipc_post_rx32_mode = 0)
received IPC_MSGTYPE_REBOOT_ACK from fid d020
received IPC_MSGTYPE_FILE_TX_START: status = 0, tx_id = 3, rec_id = 1, more = 1, tx_window_size = 20
src fname = monitor.sig, length = 256, checksum = 77b3, dst fname = monitor.sig, save_to_mem = 0, copy_to_boot = 0
Write to memory done.
download_write_to_flash: fp = 1, length 256
done.
MP Synchronization for monitor.sig is done
received IPC_MSGTYPE_FILE_TX_START: status = 0, tx_id = 4, rec_id = 1, more = 1, tx_window_size = 20
src fname = primary.sig, length = 256, checksum = 57c7, dst fname = primary.sig, save_to_mem = 0, copy_to_boot = 0
Write to memory done.
download_write_to_flash: fp = 1, length 256
done.
MP Synchronization for primary.sig is done
sys_ipc_active_boot_info: bp - board class: 0x84, chassis type 0xf0.
sys_ipc_active_boot_info: mp - board class: 0x81, chassis type 0x6b.
INFO: New 16-slot backplane is detected.
BOOT INFO: load image from primary copymy_slot = 34
rw_get_mac_and_arbitration_delay: peer is active, give up.
Total Sector = 4098528
Total size   = 2098M bytes
Storage Card SLOT INFO:SILICONSYSTEMS INC 2GB-3625
Slot 1 - File System init done

Total Sector = 4098528
Total size   = 2098M bytes
Storage Card SLOT INFO:SILICONSYSTEMS INC 2GB-3625
Slot 2 - File System init done

load_monitor_config: BP board_class=0x84, chassis_type=0xf0
load_monitor_config: MP board_class=0x81, chassis_type=0x6b
INFO: New 16-slot backplane is detected.

keep_alive_enabled = 00000001
hb_failure_reset = 00000001
fsm_no_timeout = 00000000
scp_debug = 00000000
scp_debug_lacp = 00000000
scp_debug_port = 1(0)
scp_debug_port = 1(0)

Chassis and Module type mismatch check enabled (g_chassis_type_check_enabled = 1)
3: RW_MBRIDGE_STATUS_REG = 07ff3fff
g_max_slave_slot = 16
my_slot = 34
active_mp_slot = 33, standby_mp_slot = 34
Hit any key to skip IPC handshake with Active Management...
Standby Management Module

#####################################################
About https://extremeportal.force.com/ExtrArticleDetail?an=000063714
I believe I ran into this but I just did an "erase" (deleted all files from secondary MP flash) and secondary MP flashed successfully into... well, MR2-X.

Re: BR-MLX-MR2-M became an X card after firmware update

joergkost — Sat, 18 Dec 2021 18:13:07 GMT

Oh, strange that it shows board-class 0x81. I think the board class should be 0x1

Could you run
hw pid slot standby
hw pid show
hw pid slot mgmt
hw pid show

Re: BR-MLX-MR2-M became an X card after firmware update

theale — Sat, 18 Dec 2021 19:24:54 GMT

Hi Aleksandr,

There is a known issue when attempting to install a standby management module (MP), the issue will only be seen when the active MP is running version 6.2 or above, and the newly inserted standby MP is loaded with version 6.1 or below.

Please check the article in the link below to see if you are hitting this issue and then follow the recovery procedure:

https://extremeportal.force.com/ExtrArticleDetail?an=000063714

Note:
The apparent change in module type is probably incorrect, as it has not synced and booted correctly this is not displaying the correct module type and should correct itself when the module is able to boot into the standby state.

Re: BR-MLX-MR2-M became an X card after firmware update

Aleksandr — Sun, 19 Dec 2021 11:39:42 GMT

to joergkost:
hw pid slot standby
hw pid show

=== Manufacturing Information ====
Board Class : 0x01 (Mgmt)
Factory Assembly Part Number : 60-1002374-05
Chassis Type: BR-MLX-MR2-X (0xae)
--Factory Serial Number : BVP0447G03F
Date of Manufacture : 23-3-2012
Bench Test Status : PASSED
Burn-in Test Status : PASSED
Manufacturing Deviation :
RMA Number :
PCB Revision :
MFG Test :

hw pid slot mgmt
hw pid show

=== Manufacturing Information ====
Board Class : 0x01 (Mgmt)
Factory Assembly Part Number : 60-1002374-07
Chassis Type: BR-MLX-MR2-M (0x6b)
--Factory Serial Number : BVP0419L02Z
Date of Manufacture : 19-9-2015
Bench Test Status : PASSED
Burn-in Test Status : PASSED
Manufacturing Deviation :
RMA Number :
PCB Revision :
MFG Test :

Re: BR-MLX-MR2-M became an X card after firmware update

Aleksandr — Sun, 19 Dec 2021 11:42:37 GMT

to theale :

Are you suggesting that I should boot that module alone in the system (as primary and alone) ?

Re: BR-MLX-MR2-M became an X card after firmware update

theale — Sun, 19 Dec 2021 19:08:28 GMT

Hi Aleksandr,

No I am not suggesting that you boot the new MP module alone in the system, did you read the article in the link that I provided, nowhere does it talk about making the new standby MP the active MP.
If you want to do that you would have to upgrade the management module from a tftp server, it is far easier to follow the procedure in the link that I provided and will not interrupt the operation of your router.

The procedure is basically as follows:
1) Check that messages like the following are scrolling on the active MP console.
Error:update_standby_mp_image_info: num_lp_image_info (9556573) is larger than LP_IMAGE_TYPE_NUMBER (1)
2) Reboot the standby MP
3) On the standby MP console press the b key several times to interrupt the boot sequence and put the module into monitor mode.
4) On the standby MP console run the dir command to list files.
5) Compare the file size with the number in parentheses from the error messages on the active MP console.
6) Delete the file that matches that file size, probably the lp-primary-0 file, with the del command.
7) Reboot the standby MP from monitor mode with the reset command
😎 After the module has completed file sync and booted into the standby role check if the monitor version matches the active MP.
9) If the monitor versions do not match reboot the standby MP again using the command reboot-standby from the active MP.
10) After the standby MP has booted check if the monitor version matches the active MP, if it does you have completed the recovery procedure.

This whole procedure should only take a few minutes to complete, so check the active MP console for the error messages to confirm you are seeing this issue, then follow the recovery procedure to correct it.

Re: BR-MLX-MR2-M became an X card after firmware update

Aleksandr — Sun, 19 Dec 2021 19:58:56 GMT

To theale :

In my first answer to joergkost I wrote following:

"About https://extremeportal.force.com/ExtrArticleDetail?an=000063714
I believe I ran into this but I just did an "erase" (deleted all files from secondary MP flash) and secondary MP flashed successfully into... well, MR2-X."

So even If I had this issue it's not the case at the moment. Both cards are running same version of firmware (boot/monitor/os) now and I don't have this error(from the article 000063714) in the message log
Nevertheless I did what you asked for and still have same error " Standby management module is not enabled due to type mismatch
" and MR2-M card that thinks that it is MR2-X

show ver + dir from module in question (standby module with identity crisit)
MP-2 Monitor>show vers
Boot     05.09.00T165     Mar 19 2015 03:16:46 PDT xmprm05900
Monitor 06.02.00T165     Aug 17 2017 11:22:12 PDT xmb06200 (code)
1.666 GHz Power PC processor (version 8004/0202) 166.6667 MHz bus
started=cold start
MP-2 Monitor>dir
   660145 [c395] ___mbridge
   573366 [0000] lp-monitor-0
  9567381 [0000] lp-primary-0
   546965 [0000] monitor
      256 [77b3] monitor.sig
10676645 [0000] primary
      256 [57c7] primary.sig
     4438 [edf5] startup-config
22029452 bytes 8 File(s)
109314048 bytes free

Same from main MP
SL M1: BR-MLX-MR2-M Management Module Active (Serial #: BVP0419L02Z, Part #: 60-1002374-07):
Boot     : Version 5.9.0T165 Copyright (c) 1996-2015 Brocade Communications Systems, Inc.
Compiled on Mar 19 2015 at 03:16:46 labeled as xmprm05900
(521771 bytes) from boot flash
Monitor : Version 6.2.0T165 Copyright (c) 1996-2015 Brocade Communications Systems, Inc.
Compiled on Aug 17 2017 at 11:22:12 labeled as xmb06200
(546965 bytes) from code flash
IronWare : Version 6.2.0bT163 Copyright (c) 1996-2015 Brocade Communications Systems, Inc.
Compiled on Apr 4 2018 at 08:59:36 labeled as xmr06200b
(10676645 bytes) from Primary
Board ID : 00 MBRIDGE Revision : 37
1666 MHz Power PC processor 7448 (version 8004/0202) 166 MHz bus
512 KB Boot Flash (MX29LV040C), 128 MB Code Flash (MT28F256J3)
4096 MB DRAM INSTALLED
4096 MB DRAM ADDRESSABLE
Active Management uptime is 3 days 21 hours 16 minutes 45 seconds

dir
Directory of /flash/

05/24/2017 19:10:38                       1 $$snmp_boots
08/24/2021 18:09:07                   1,564 $$sshrsahost.key
08/24/2021 18:40:12                   3,588 $$user_profile
12/14/2021 11:14:33                 660,145 ___mbridge
01/18/2021 13:45:19                     478 boot.sig
12/14/2021 10:29:26                     460 fips_public_key.crt
01/18/2021 13:49:30                 524,288 lp-boot
01/18/2021 13:49:28                 524,288 lp-boot-0
01/18/2021 13:45:26                     478 lp-boot.sig
12/14/2021 10:29:39                     256 lp-mon.sig
12/14/2021 11:07:44                 573,366 lp-monitor-0
12/14/2021 10:31:59                     256 lp-pri.sig
12/14/2021 11:09:38               9,567,381 lp-primary-0
12/14/2021 10:33:54                     256 lpfpga.sig
12/14/2021 10:29:26                   3,011 manifest
12/14/2021 10:29:26                     256 manifest.sig
12/14/2021 11:07:27                     256 mbridge.sig
12/14/2021 12:00:54                 546,965 monitor
12/14/2021 10:29:26                     256 monitor.sig
12/14/2021 11:08:48              10,676,645 primary
12/14/2021 10:29:51                     256 primary.sig
12/15/2021 23:00:28                   4,438 startup-config

###################################################################

Could it be that by running "erase" on module in question I accidentally removed some file from flash that was telling card what it is(its identity)?
I believe these cards (mr2-x and mr2-m) are the same in hardware and it's just a marketing trick, just a software difference. So it's a question for certain bits in the right place (that will tell card what it should be)

Re: BR-MLX-MR2-M became an X card after firmware update

joergkost — Mon, 10 Jan 2022 15:08:02 GMT

I tried this in the lab today, but unfortunately, I cannot reproduce it. I am going from 5.8 to 6.2; then, "erase flash" of the second management board. It always drops me back to the two -M versions. The best possibility for you may do the RMA/support request ticket or get a new -M version. EPROM shall take the identity, and they may be intelligent procedures to reprogram. However, I don't know these.

M1 (left ):BR-MLX-MR2-M Management Module Standby(App sync/Stabilize state)
M2 (right):BR-MLX-MR2-M Management Module Active