topic RE: HELP! Stack unexpected reboot in ExtremeSwitching (Other)

HELP! Stack unexpected reboot

Mykhaylo_Skrypk — Wed, 18 Jan 2017 21:34:00 GMT

XOS: ExtremeXOS version 15.3.3.5 v1533b5-patch1-2

One of our extreme x460/x450 stacks rebooted unexpectedly this morning (at 04:52 ) Logs suggests the following:

2017-01-18 04:57:14.33 Stacking port 1:1 link up at 10Gbps.
2017-01-18 04:57:13.99 Starting hal initialization ....
2017-01-18 04:57:12.29 telnetd listening on port 23

2017-01-18 04:57:06.18 The stack MAC address is not correctly configured on this node. The stack can not operate properly in this condition. Please correct and reboot.
2017-01-18 04:57:03.16 DM started
2017-01-18 04:57:02.95 The Node Manager (NM) has started processing.
2017-01-18 04:57:02.15 EPM Started
2017-01-18 04:57:01.83 Changing to watchdog warm reset mode
2017-01-18 04:52:20.87 Slot-1 FAILED (1) Backup lost
2017-01-18 04:52:20.83 Shutting down all processes
2017-01-18 04:52:20.53 Node State[4] = FAIL (Backup lost)
2017-01-18 04:52:20.53 MASTER decided that I am not BACKUP anymore
2017-01-18 04:52:20.53 BACKUP NODE (Slot-1) DOWN

has anyone had a similar problem?
Thx,
Mykhaylo

RE: HELP! Stack unexpected reboot

Aleixo_Gomes — Wed, 18 Jan 2017 21:43:00 GMT

show stacking detail , will provide the info on configured stack mac -address
show ports stack-ports rxerrors no-refresh
show ports stack-ports txerrors no-refresh
above two commands will provide if there are any crc errors on stack ports , if they re incrementing then consider , swapping stack cables or reseating stack ports connection.

RE: HELP! Stack unexpected reboot

Vellachery__Sum — Wed, 18 Jan 2017 21:43:00 GMT

Mykhaylo,

Did you get a chance to review the logs from Slot-1? (during that time-stamp around 4:52)
Copy paste the output of "show debug system dump" to see if there is any process crash?
Also, check if there are any process crash files in the internal-memory? "ls internal-memory"

RE: HELP! Stack unexpected reboot

Mykhaylo_Skrypk — Wed, 18 Jan 2017 21:43:00 GMT

-> show debug system-dump slot 1 ===============================================
Slot-1 system dump information
===============================================
core_dump_info storage: 8/3072 used [empty]
failure: kernel oops
reason: NMI1
time: Wed Apr 22 05:00:13 2015
where: extraps_handle_nmi:286
$0 : z0=00000000 at=ffefffff v0=00100000 v1=40100028
$4 : a0=00000001 a1=1000dc01 a2=00000001 a3=00000000
$8 : t0=80000380 t1=00080000 t2=80890a50 t3=2adff000
$12: t4=80000000 t5=00000000 t6=c2006448 t7=00440000
$16: s0=00001d50 s1=808b6ef0 s2=817fc900 s3=00002000
$20: s4=00000001 s5=00100000 s6=808b0000 s7=ffffffbf
$24: t8=004444a4 t9=2ad32f04
$28: gp=86b20000 sp=86b21b60 s8=808c0000 ra=80299c5c
Hi : 00000357
Lo : 00000000
epc : 80299d00 flush_all_zero_pkmaps+0x1d4/0x1ec Tainted: P
Status: 504800c0
Cause : 90800000
86b21b60: 86b19b60 00000000 001200d2 00000000 80630000 808b6048 817f3360 808b0000
86b21b80: 86b21ba8 8060baa0 80634d00 00000002 808b0000 80299ec8 00000a38 80292404
86b21ba0: 883bc624 000000d8 000000d8 ffffffef 00000003 8028824c 817f3360 883bc624
86b21bc0: ffffffff 000000d8 817f3360 8f3b2e00 883bc54c 000d8000 883bc624 86b21cf0
86b21be0: 00001000 00000000 836b3e00 802249fc 00000000 001fffff 00000000 80288474
86b21c00: 883bc54c 000d7000 8f3b2e00 80346ffc 83580718 8034d5e8 000d8000 883bc57c
86b21c20: 00000000 000000d8 883bc57c 86b21cf0 00001000 817f3360 883bc57c 803471e8
86b21c40: fe3ff000 00000000 80630000 817fada0 883bc57c 80299a70 883bc57c 00000000
86b21c60: 00000000 00001000 83580718 817fada0 883bc57c 80346f68 00000000 80564378
86b21c80: 883bc624 86b21cf0 000d7000 00001000 86b21c98 00000003 00001000 86b21cf0
86b21ca0: 000d7000 00001000 00000000 000d8000 00001000 00000000 80564378 883bc624
86b21cc0: 86b21cf0 802892a8 807c37e0 81a0a7e0 86b21e20 86b21e18 00001000 00000000
86b21ce0: 86b21ce8 86b21cec 817f3360 802873bc 86b21e18 00000001 00000000 00001000
86b21d00: 00000000 00000000 883bc624 883bc57c 000d8000 00000000 80564378 802c7b0c
86b21d20: 883bc57c 836b3e00 883bc624 86b21e70 86b21e20 86b21e18 86b21dc8 00000000
86b21d40: 000d8000 80289b98 86b21e20 80213920 00000000 86d96000 00000000 000d8000
log: ... PC 80299d10(flush_all_zero_pkmaps+0x1e4/0x1ec) at 63 seconds.
log: <0>
log: <0>CPU1: NMI at 0x80299d00, liveness (1763835910, 263766010)

Version #0 SMP Tue Jan 7 13:12:45 EST 20 by release-manager@biltmore.extrem Release 2.6.28.9cougar
Call Trace:
@[<80299d00>] flush_all_zero_pkmaps+0x1d4/0x1ec
@[<80299ec8>] kmap_high+0x1b0/0x25c
@[<802249fc>] __kmap+0x60/0x84
@[<80346ffc>] jffs2_do_readpage_nolock+0x3c/0xf4
@[<803471e8>] jffs2_write_begin+0x134/0x334
@[<802892a8>] generic_file_buffered_write+0x124/0x38c
@[<80289b98>] __generic_file_aio_write_nolock+0x2fc/0x60c
@[<8028a35c>] generic_file_aio_write+0x78/0x12c
@[<802b12b4>] do_sync_write+0xe0/0x124
@[<802b1e00>] vfs_write+0xb4/0x158
@[<802b1f9c>] sys_write+0x4c/0xa4
@[<80221b58>] stack_done+0x20/0x3c

[<80292404>] ____pagevec_lru_add+0x1d4/0x210
[<8028824c>] add_to_page_cache_locked+0x7c/0x10c
[<80288474>] grab_cache_page_write_begin+0xe0/0x10c
[<8034d5e8>] jffs2_write_inode_range+0x260/0x37c
[<80299a70>] kunmap_high+0x28/0xe4
[<80346f68>] jffs2_write_end+0x248/0x2a0
[<802873bc>] file_remove_suid+0x1c/0x9c
[<802c7b0c>] file_update_time+0x58/0x140
[<80213920>] ret_from_irq+0x0/0x4
Build directory: /data2/release-manager/v15_3_3_5-patch1-2/summit_rmi

Stacktrace:
reason: NMI0
id: 1429675213
$0 : z0=00000000 at=1000dc00 v0=805f2000 v1=8055bdd0
$4 : a0=00000000 a1=86b18920 a2=00000000 a3=86b18958
$8 : t0=80000380 t1=00080000 t2=00000000 t3=0000657f
$12: t4=9ae88a24 t5=00000001 t6=00000000 t7=2c652000
$16: s0=86b18920 s1=81a06920 s2=00000001 s3=00000000
$20: s4=0000000f s5=807c3920 s6=807c5e98 s7=00000000
$24: t8=01243000 t9=81a06920
$28: gp=807c4000 sp=807c5e88 s8=3030249e ra=80237fb0
Hi : 00095eff
Lo : dc887e00
epc : 80232778 resched_task+0x4/0xb0 Tainted: P
Status: 504800c0
Cause : 10800000
807c5e88: 002840cb 302ea2d4 807c5eb8 fffffbff 1000dc00 807c5ef0 1f72a3ea 86b25a60
807c5ea8: 81a06318 81a06350 86b25a60 81a06350 002840cb 3030249e 00000001 8025ccc8
807c5ec8: 00000000 81a06350 00000002 00000001 86b25a60 8025cd98 00000001 c4959b18
807c5ee8: 00000000 fffffbff 002840cb 3030249e 00000000 81a06318 3b9aca00 8025e250
807c5f08: ffffffff 86c12000 00000000 c4a4c454 002840cb 3030249e 7fffffff ffffffff
807c5f28: 7fffffff ffffffff 002840cb 81a06320 80844014 00000000 00000007 00000001
807c5f48: 00000000 807c2000 80844014 00001da0 00000000 8021b870 1000dc03 00000008
807c5f68: c26cd594 2ad83894 80608e70 80271b40 806299c0 00000022 806299f4 886348a0
807c5f88: 80628fa0 00000007 80628fd4 81a0d000 80878b00 80271c68 00000000 00000000
807c5fa8: 00000000 00000000 00000000 807c5fd0 00000000 00004000 81a0d000 80878b00
807c5fc8: 807c2000 80213b44 00000000 00000000 00000000 00000000 00000000 00000000
807c5fe8: 00000000 00000000 00000000 807c4000 805f3de8 802156e0 1bad2bad 1bad2bad
807c6008: 1bad2bad 1bad2bad 1bad2bad 1bad2bad 1bad2bad 1bad2bad 1bad2bad 1bad2bad
807c6028: 1bad2bad 1bad2bad 1bad2bad 1bad2bad 1bad2bad 1bad2bad 1bad2bad 1bad2bad
807c6048: 1bad2bad 1bad2bad 1bad2bad 1bad2bad 1bad2bad 1bad2bad 1bad2bad 1bad2bad
807c6068: 1bad2bad 1bad2bad 1bad2bad 1bad2bad 1bad2bad 1bad2bad 1bad2bad 1bad2bad
Call Trace:
@[<80232778>] resched_task+0x4/0xb0
@[<80237fb0>] try_to_wake_up+0x12c/0x2d0
@[<8025ccc8>] hrtimer_wakeup+0x1c/0x2c
@[<8025cd98>] __run_hrtimer+0xc0/0xd4
@[<8025e250>] hrtimer_interrupt+0x258/0x384
@[<8021b870>] c0_compare_interrupt+0x50/0xa0
@[<80271b40>] handle_IRQ_event+0x78/0xe4
@[<80271c68>] __do_IRQ+0xbc/0x264
@[<80213b44>] call_handle_irq+0x20/0x3c
@[<802156e0>] do_IRQ+0x9c/0x138
@[<80213920>] ret_from_irq+0x0/0x4
@[<80275bc0>] rcu_pending+0xb8/0x144
@[<80216510>] cpu_idle+0x30/0x90

RE: HELP! Stack unexpected reboot

Vellachery__Sum — Wed, 18 Jan 2017 21:43:00 GMT

Mykhaylo,

The kernel oops which we have in the dump information has been occurred on Wed Apr 22 05:00:13 2015. Hence, this has nothing to do with the
current unexpected reboot.

As my colleague Aleixo suggested earlier. Please provide the output of
show stacking detail , will provide the info on configured stack mac -address
show switch from the Master node
show stacking
show ports stack-ports rxerrors no-refresh
show ports stack-ports txerrors no-refresh

RE: HELP! Stack unexpected reboot

Vellachery__Sum — Wed, 18 Jan 2017 21:43:00 GMT

Adding to it, please provide the output of "show slot".
Let's check if there are any Rx/Tx errors in the stacking ports. In case, if they are getting incremented we try to swap the stacking
cable

RE: HELP! Stack unexpected reboot

Mykhaylo_Skrypk — Wed, 18 Jan 2017 21:43:00 GMT

Slot-1 es-vil-vbwc-20.7 # show slot

Slots Type Configured State Ports

--------------------------------------------------------------------

Slot-1 X460-48p X460-48p Operational 58

Slot-2 X460-48p X460-48p Operational 58

Slot-3 X450e-48p X450e-48p Operational 50

Slot-4 X460-48p X460-48p Operational 58

Slot-5 X450e-48p X450e-48p Operational 50

Slot-6 Empty 0

Slot-7 Empty 0

Slot-8 Empty 0

Slot-1 es-vil-vbwc-20.9 # show stacking

Stack Topology is a Ring

Active Topology is a Ring

Node MAC Address Slot Stack State Role Flags

------------------ ---- ----------- ------- ---

*00:04:96:83:4c:a6 1 Active Master CA-

00:04:96:83:4c:a8 2 Active Backup CA-

00:04:96:36:a2:53 3 Active Standby CA-

00:04:96:83:4c:2e 4 Active Standby CA-

00:04:96:35:cf:25 5 Active Standby CA-

* - Indicates this node

Flags: (C) Candidate for this active topology, (A) Active Node

(O) node may be in Other active topology

Thank guys. Just getting :

show ports stack-ports rxerrors no-refresh

show ports stack-ports txerrors no-refresh

RE: HELP! Stack unexpected reboot

Mykhaylo_Skrypk — Wed, 18 Jan 2017 21:43:00 GMT

Sorry as l don't have access to the system dyrectly cannot provide a nice reply/output requested.

RE: HELP! Stack unexpected reboot

Vellachery__Sum — Wed, 18 Jan 2017 21:43:00 GMT

Mykhaylo,

You can share the outputs, once you have access to the switch. We will take a look at it and would share our feedback accordingly....

RE: HELP! Stack unexpected reboot

Mykhaylo_Skrypk — Wed, 18 Jan 2017 21:43:00 GMT

Looks like no errors:

Slot-1 es-vil-vbwc-20.3 # show switch | i nt.TCurrent Time: Wed Jan 18 17:46:43 2017
Slot-1 es-vil-vbwc-20.4 # show stacking detail
Stacking Node 00:04:96:83:4c:a6 information:
Current:
Stacking : Enabled
Role : Master
Priority : Automatic
Slot number : 1
Stack state : Active
Master capable? : Yes
Stacking protocol : Standard
License level restriction :
In active topology? : Yes
Factory MAC address : 00:04:96:83:4c:a6
Stack MAC address : 02:04:96:83:4c:a6
Alternate IP address :
Alternate gateway :
Stack Port 1:
State : Operational
Blocked? : No
Control path active? : Yes
Selection : Native
Stack Port 2:
State : Operational
Blocked? : No
Control path active? : Yes
Selection : Native
Configured:
Stacking : Enabled
Master capable? : Yes
Slot number : 1
Stack MAC address : 02:04:96:83:4c:a6
Stacking protocol : Standard
License level restriction :
Stack Port 1:
Selection : Native
Stack Port 2:
Selection : Native

Stacking Node 00:04:96:83:4c:a8 information:
Current:
Stacking : Enabled
Role : Backup
Priority : Automatic
Slot number : 2
Stack state : Active
Master capable? : Yes
Stacking protocol : Standard
License level restriction :
In active topology? : Yes
Factory MAC address : 00:04:96:83:4c:a8
Stack MAC address : 02:04:96:83:4c:a6
Alternate IP address :
Alternate gateway :
Stack Port 1:
State : Operational
Blocked? : No
Control path active? : Yes
Selection : Native
Stack Port 2:
State : Operational
Blocked? : No
Control path active? : Yes
Selection : Native
Configured:
Stacking : Enabled
Master capable? : Yes
Slot number : 2
Stack MAC address : 02:04:96:83:4c:a6
Stacking protocol : Standard
License level restriction :
Stack Port 1:
Selection : Native
Stack Port 2:
Selection : Native

Stacking Node 00:04:96:36:a2:53 information:
Current:
Stacking : Enabled
Role : Standby
Priority : Automatic
Slot number : 3
Stack state : Active
Master capable? : Yes
Stacking protocol : Standard
License level restriction :
In active topology? : Yes
Factory MAC address : 00:04:96:36:a2:53
Stack MAC address : 02:04:96:83:4c:a6
Alternate IP address :
Alternate gateway :
Stack Port 1:
State : Operational
Blocked? : No
Control path active? : Yes
Selection : Native
Stack Port 2:
State : Operational
Blocked? : No
Control path active? : Yes
Selection : Native
Configured:
Stacking : Enabled
Master capable? : Yes
Slot number : 3
Stack MAC address : 02:04:96:83:4c:a6
Stacking protocol : Standard
License level restriction :
Stack Port 1:
Selection : Native
Stack Port 2:
Selection : Native

Stacking Node 00:04:96:83:4c:2e information:
Current:
Stacking : Enabled
Role : Standby
Priority : Automatic
Slot number : 4
Stack state : Active
Master capable? : No
Stacking protocol : Standard
License level restriction :
In active topology? : Yes
Factory MAC address : 00:04:96:83:4c:2e
Stack MAC address : 02:04:96:83:4c:a6
Alternate IP address :
Alternate gateway :
Stack Port 1:
State : Operational
Blocked? : No
Control path active? : Yes
Selection : Native
Stack Port 2:
State : Operational
Blocked? : Yes
Control path active? : Yes
Selection : Native
Configured:
Stacking : Enabled
Master capable? : No
Slot number : 4
Stack MAC address : 02:04:96:83:4c:a6
Stacking protocol : Standard
License level restriction :
Stack Port 1:
Selection : Native
Stack Port 2:
Selection : Native

Stacking Node 00:04:96:35:cf:25 information:
Current:
Stacking : Enabled
Role : Standby
Priority : Automatic
Slot number : 5
Stack state : Active
Master capable? : Yes
Stacking protocol : Standard
License level restriction :
In active topology? : Yes
Factory MAC address : 00:04:96:35:cf:25
Stack MAC address :
Alternate IP address :
Alternate gateway :
Stack Port 1:
State : Operational
Blocked? : Yes
Control path active? : Yes
Selection : Native
Stack Port 2:
State : Operational
Blocked? : No
Control path active? : Yes
Selection : Native
Configured:
Stacking : Enabled
Master capable? : Yes
Slot number : 5
Stack MAC address :
Stacking protocol : Standard
License level restriction :
Stack Port 1:
Selection : Native
Stack Port 2:
Selection : Native
Slot-1 es-vil-vbwc-20.5 #
Slot-1 es-vil-vbwc-20.5 #
Slot-1 es-vil-vbwc-20.5 # show ports stack-ports rxerrors no-refresh
Port Rx Error Monitor
Port Link Rx Rx Rx Rx Rx Rx Rx
State Crc Over Under Frag Jabber Align Lost
================================================================================
1:1 A 0 0 0 0 0 0 0
1:2 A 12 0 0 0 0 0 0
2:1 A 0 0 0 0 0 0 0
2:2 A 0 0 0 0 0 0 0
3:1 A 0 0 0 0 0 0 0
3:2 A 0 0 0 0 0 0 0
4:1 A 0 0 0 0 0 0 0
4:2 A 0 0 0 0 0 0 0
5:1 A 0 0 0 0 0 0 0
5:2 A 0 0 0 0 0 0 0
================================================================================
Link State: A-Active, R-Ready, NP-Port Not Present L-Loopback
Slot-1 es-vil-vbwc-20.6 # show ports stack-ports txerrors no-refresh
Port Tx Error Monitor
Port Link Tx Tx Tx Tx Tx Tx
State Coll Late coll Deferred Errors Lost Parity
================================================================================
1:1 A 0 0 0 0 0 0
1:2 A 0 0 0 0 0 0
2:1 A 0 0 0 0 0 0
2:2 A 0 0 0 0 0 0
3:1 A 0 0 0 0 0 0
3:2 A 0 0 0 0 0 0
4:1 A 0 0 0 0 0 0
4:2 A 0 0 0 0 0 0
5:1 A 0 0 0 0 0 0
5:2 A 0 0 0 0 0 0
================================================================================
> indicates Port Display Name truncated past 8 characters
Link State: A-Active, R-Ready, NP-Port Not Present L-Loopback
Slot-1 es-vil-vbwc-20.7 #
Slot-1 es-vil-vbwc-20.7 # show switch | i nt.T
Current Time: Wed Jan 18 17:47:10 2017
Slot-1 es-vil-vbwc-20.8 #

RE: HELP! Stack unexpected reboot

Mykhaylo_Skrypk — Wed, 18 Jan 2017 21:43:00 GMT

Would be nice to have an option attaching .txt files while writing a reply as output takes a lot of screen space! Just a thought

RE: HELP! Stack unexpected reboot

Vellachery__Sum — Wed, 18 Jan 2017 21:43:00 GMT

Mykhaylo,

Have an eye on the stack-ports rxerrors. As of now i only see CRC errors on stacking port 1:2 as shown below.

Try clearing the counters and monitor the rxerrors on the stacking ports. In case, if you notice them getting frequently.
incremented. Swap the stacking cables and monitor them once again.

RE: HELP! Stack unexpected reboot

Vellachery__Sum — Wed, 18 Jan 2017 21:43:00 GMT

Mykhaylo,

Adding to it, did you get an opportunity to review the logs from Slot-1 at the time of issue occurrence?

Also, ensure that we rule out physical layer related issues (isolating the power cord/power source).

RE: HELP! Stack unexpected reboot

Mykhaylo_Skrypk — Wed, 18 Jan 2017 21:43:00 GMT

Yeah seen these 12 CRC errors and will do as advised. Thanks. The logs are actually already from the slot 1. Because when i am trying to telnet to slot 1 getting expected error: Error: Cannot establish connection to self.

RE: HELP! Stack unexpected reboot

Mykhaylo_Skrypk — Wed, 18 Jan 2017 21:43:00 GMT

Swapping stack cable and reseating stack port (1:2) connection seems to resolved CRC errors

RE: HELP! Stack unexpected reboot

Patrick_Voss — Wed, 18 Jan 2017 21:45:00 GMT

If you could also provide a "show version" output.

RE: HELP! Stack unexpected reboot

Mykhaylo_Skrypk — Wed, 18 Jan 2017 21:45:00 GMT

version 15.3.3.5 v1533b5-patch1-2

RE: HELP! Stack unexpected reboot

Patrick_Voss — Wed, 18 Jan 2017 21:45:00 GMT

Can I get the full "show version" output?

RE: HELP! Stack unexpected reboot

Mykhaylo_Skrypk — Wed, 18 Jan 2017 21:45:00 GMT

-> show version
Slot-1 : 800324-00-10 1xxxN-4xx5 Rev 10.0 BootROM: 2.0.1.7 IMG: 15.3.3.5
Slot-2 : 800324-00-10 xxxN-40xx4 Rev 10.0 BootROM: 2.0.1.7 IMG: 15.3.3.5
Slot-3 : 800307-00-01 xxxx-80xx2 Rev 1.0 BootROM: 1.0.5.5 IMG: 15.3.3.5
Slot-4 : 800324-00-10 xxxxxN-xx919 Rev 10.0 BootROM: 2.0.1.7 IMG: 15.3.3.5
Slot-5 : 800190-00-07 xxxxx-xx140 Rev 7.0 BootROM: 1.0.5.5 IMG: 15.3.3.5

RE: HELP! Stack unexpected reboot

Mykhaylo_Skrypk — Wed, 18 Jan 2017 21:56:00 GMT

WOW! That is impressive community response 🙂

So l need to telnet to slot 1 and get logs from there? show debug system dump and internal-memory. What is the exact command as l can only use show debug system

Thx all,
Mykhaylo