RFS7000 Crashing - Wing 5.8.5

Phil_storey — Wed, 20 Sep 2017 13:43:00 GMT

Hi
over the last couple of day the RFS7K has dropped off the network and stops all MU form connecting. The WIF is still visable ( Ap's are 7131 & 7532 ) but no connection can be made.
I have looked on the Primary unit and there was some crash files about 6 in total but dating from earlier in the this month.

But its all a bit meaningless to me:

Thu Sep 7 10:39:35 2017: /home/spin/workspace/ncap_br2/build_br2/obj/qs5/src/sys/cfgd/remote_debug/server.py:964 server_tasklet() session args = {'xfer_args': {'tzsp_encap': True}, 'output_type': 9}
Return success for start_session

Thu Sep 7 10:39:35 2017: /home/spin/workspace/ncap_br2/build_br2/obj/qs5/src/sys/cfgd/remote_debug/server.py:1034 session_initiator_tasklet() requesting host 70.81.BE.8E
Thu Sep 7 10:39:35 2017: /home/spin/workspace/ncap_br2/build_br2/obj/qs5/src/sys/cfgd/remote_debug/client.py:778 req_listen_tasklet() Peer 70.81.BE.8E connected over seqpacket port 36

Thu Sep 7 10:39:35 2017: /home/spin/workspace/ncap_br2/build_br2/obj/qs5/src/sys/cfgd/remote_debug/server.py:1066 session_initiator_tasklet() notified 1 of 1 hosts

In handle_request_tasklet =====> req message
Thu Sep 7 10:39:35 2017: /home/spin/workspace/ncap_br2/build_br2/obj/qs5/src/sys/cfgd/remote_debug/client.py:734 handle_request_tasklet() Got request 3
Thu Sep 7 10:39:35 2017: /home/spin/workspace/ncap_br2/build_br2/obj/qs5/src/sys/cfgd/remote_debug/client.py:182 handle_live_pktcap_request() handle_live_pktcap_request from (70.81.BE.8E, 54038)

How ever when I looked on the backup unit there were loads more like this

Thu Sep 7 10:45:28 2017: /home/spin/workspace/ncap_br2/build_br2/obj/qs5/src/sys/cfgd/idpd/role.py:193 update_acl_to_dp() [role def Guest] ACTION: Update out acl for DP
Thu Sep 7 10:45:28 2017: /home/spin/workspace/ncap_br2/build_br2/obj/qs5/src/sys/cfgd/idpd/role.py:237 update_acl_to_dp() [role def Guest] No ip out acl configured, applying default
Thu Sep 7 10:45:28 2017: /home/spin/workspace/ncap_br2/build_br2/obj/qs5/src/sys/cfgd/idpd/role.py:239 update_acl_to_dp() [role def Guest] Updating DEFAULT acl to DP
Thu Sep 7 10:45:28 2017: /home/spin/workspace/ncap_br2/build_br2/obj/qs5/src/sys/cfgd/idpd/role.py:237 update_acl_to_dp() [role def Guest] No ipv6 out acl configured, applying default
Thu Sep 7 10:45:28 2017: /home/spin/workspace/ncap_br2/build_br2/obj/qs5/src/sys/cfgd/idpd/role.py:239 update_acl_to_dp() [role def Guest] Updating DEFAULT acl to DP

There was also a panic file dated today- it seems to be complaining about a soft lockup

NUM_DATA_THREADS = 258048
<6>highmem remaining: 19023KB/288MB (6%)
<6>module_alloc:58 (size 169360)
<6>module_alloc:58 (size 128)
<6>Dataplane Hardware Services Layer for IP Infusion
<6>NET: Registered protocol family 48
<6>NET: Registered protocol family 40
<6>NET: Registered protocol family 42
<6>module_alloc:58 (size 26240)
<6>Compact Flash Driver Module Installed
<6>ge1 { no link }
<6>me1: DSPCFG accepted after 0 usec.
<6>ADDRCONF(NETDEV_UP): me1: link is not ready
<6>ge1 { autoneg'ed 1Gb/s, autoneg'ed full duplex, Cu }
<6>me1: DSPCFG accepted after 0 usec.
<6>ge1 { no link }
<6>ge1 { autoneg'ed 1Gb/s, autoneg'ed full duplex, Cu }
<6>ge1 { no link }
<3>BUG: soft lockup detected on CPU#0!
<6>Cpu 0
<6>$ 0 : 00000000 50104c00 00000008 0000000c
<6>$ 4 : 00000000 88171cd8 88171c98 00000000
<6>$ 8 : 510e523f e07c2654 8d140000 3ebb3f3e
<6>$12 : 8dbde1b6 00000040 88171cd0 88171cd4
<6>$16 : 8d75c1f4 8d75c1f4 80923974 88171cd8
<6>$20 : 88171c98 00000000 8d18fe30 00000000
<6>$24 : 0d1561b6 88171cbc
<6>$28 : 88170000 88171c78 000dc690 8d085928
<6>Hi : 03079680
<6>Lo : 02a6a3b0
<6>epc : 8d0856ec capture_prep_dot11+0x9ac/0xaa0 [dataplane] Tainted: PF
<6>ra : 8d085928 blake2s_Os_compress+0x148/0x480 [dataplane]
<6>Status: 50104c03 KERNEL EXL IE
<6>Cause : 00808000
<6>PrId : 000c0904
<0>Kernel panic - not syncing: softlockup: hung tasks
<6> Badness in smp_call_function at arch/mips/kernel/smp.c:159
<6>Raw Call Trace:
<6> [<80127534>] panic+0x74/0x1c0
<6> [<80127514>] panic+0x54/0x1c0
<6> [<8014c200>] watchdog+0x0/0x80
<6> [<80430000>] setup_arch+0x26c/0x514
<6> [<801081f0>] timer_interrupt+0xa0/0x248
<6> [<80430000>] setup_arch+0x26c/0x514
<6> [<8014c32c>] handle_IRQ_event+0x74/0xdc
<6> [<8014c4e8>] __do_IRQ+0x154/0x1fc
<6> [<8014c32c>] handle_IRQ_event+0x74/0xdc
<6> [<801038e0>] do_IRQ+0x1c/0x34
<6> [<80102110>] ret_from_irq+0x0/0x10
<6> [<801038e0>] do_IRQ+0x1c/0x34
<6> [<8d085928>] blake2s_Os_compress+0x148/0x480 [dataplane]
<6> [<8d0856ec>] capture_prep_dot11+0x9ac/0xaa0 [dataplane]
<6> [<8d005ee0>] dataplane_irq_add_decent_entropy_word+0x340/0x400 [dataplane]
<6> [<8012e67c>] __do_softirq+0xbc/0x1a4
<6> [<8d00610c>] dataplane_add_decent_entropy+0x16c/0x300 [dataplane]
<6> [<8d006060>] dataplane_add_decent_entropy+0xc0/0x300 [dataplane]
<6> [<8012ea14>] local_bh_enable+0x74/0xa8
<6> [<80430000>] setup_arch+0x26c/0x514
<6> [<8d13cc00>] mips_add_decent_entropy_slow+0x60/0xc0 [dataplane]
<6> [<8d125838>] target_add_decent_entropy_fast+0x178/0x1e0 [dataplane]
<6> [<8d001ba0>] config_enable_dataplane+0x120/0x940 [dataplane]
<6> [<8d001b98>] config_enable_dataplane+0x118/0x940 [dataplane]
<6> [<8017d0d8>] chrdev_open+0xc0/0x1c0
<6> [<8017d060>] chrdev_open+0x48/0x1c0
<6> [<8d079de0>] wireless_ingress_from_bridge+0x0/0x520 [dataplane]
<6> [<80185ecc>] path_lookup_open+0x18/0x24
<6> [<80170ac4>] do_sync_read+0xd4/0x160
<6> [<8d00c7ec>] append_to_queue_of_packets+0x8cc/0xc60 [dataplane]
<6> [<8016eb78>] __dentry_open+0xa8/0x1e4
<6> [<8016ed90>] nameidata_to_filp+0x4c/0x64
<6> [<80187ae8>] do_ioctl+0xb8/0xc8
<6> [<80187d74>] vfs_ioctl+0x27c/0x3e4
<6> [<8016e938>] get_unused_fd+0xd8/0x270
<6> [<80181e70>] getname+0x24/0x110
<6> [<80187f24>] sys_ioctl+0x48/0xb4
<6> [<8016ee98>] do_sys_open+0x8c/0xc0
<6> [<80101ad0>] xlr_ipi_handler+0xa8/0xb8
<6> [<8010ba8c>] stack_done+0x20/0x3c
<6> [<80102110>] ret_from_irq+0x0/0x10
<6> [<80237900>] memset_partial+0x44/0x6c
<6>
<6>Pretty Call Trace:
<6> SP 0x88171a80, RA 0x8010b514, PC 0x80109544 dump_stack+0x0/0x1c
<6>250: addiu sp,sp,-152
<6>346: frame_size 152
<6> SP 0x88171b18, RA 0x8010b514, PC 0x8010b514 smp_call_function+0x1f8/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171b30, RA 0x8010b514, PC 0x8010b514 smp_call_function+0x1f8/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171b48, RA 0x8010b514, PC 0x8010b514 smp_call_function+0x1f8/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171b60, RA 0x8010b514, PC 0x8010b514 smp_call_function+0x1f8/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171b78, RA 0x8010b514, PC 0x8010b514 smp_call_function+0x1f8/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171b90, RA 0x8010b514, PC 0x8010b514 smp_call_function+0x1f8/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171ba8, RA 0x8010b514, PC 0x8010b514 smp_call_function+0x1f8/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171bc0, RA 0x8010b514, PC 0x8010b514 smp_call_function+0x1f8/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171bd8, RA 0x8010b514, PC 0x8010b514 smp_call_function+0x1f8/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171bf0, RA 0x8010b514, PC 0x8010b514 smp_call_function+0x1f8/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171c08, RA 0x8010b514, PC 0x8010b514 smp_call_function+0x1f8/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171c20, RA 0x8010b514, PC 0x8010b514 smp_call_function+0x1f8/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171c38, RA 0x8010b514, PC 0x8010b514 smp_call_function+0x1f8/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171c50, RA 0x8010b514, PC 0x8010b514 smp_call_function+0x1f8/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171c68, RA 0x8010b514, PC 0x8010b514 smp_call_function+0x1f8/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171c80, RA 0x8010b514, PC 0x8010b514 smp_call_function+0x1f8/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171c98, RA 0x8010b514, PC 0x8010b514 smp_call_function+0x1f8/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171cb0, RA 0x8010b514, PC 0x8010b514 smp_call_function+0x1f8/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171cc8, RA 0x8010b514, PC 0x8010b514 smp_call_function+0x1f8/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171ce0, RA 0x8010b514, PC 0x8010b514 smp_call_function+0x1f8/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171cf8, RA 0x8010b514, PC 0x8010b514 smp_call_function+0x1f8/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171d10, RA 0x8010b514, PC 0x8010b514 smp_call_function+0x1f8/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171d28, RA 0x8010b514, PC 0x8010b514 smp_call_function+0x1f8/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171d40, RA 0x8010b514, PC 0x8010b514 smp_call_function+0x1f8/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6>Badness in smp_call_function at arch/mips/kernel/smp.c:160
<6>Raw Call Trace:
<6> [<80127534>] panic+0x74/0x1c0
<6> [<80127514>] panic+0x54/0x1c0
<6> [<8014c200>] watchdog+0x0/0x80
<6> [<80430000>] setup_arch+0x26c/0x514
<6> [<801081f0>] timer_interrupt+0xa0/0x248
<6> [<80430000>] setup_arch+0x26c/0x514
<6> [<8014c32c>] handle_IRQ_event+0x74/0xdc
<6> [<8014c4e8>] __do_IRQ+0x154/0x1fc
<6> [<8014c32c>] handle_IRQ_event+0x74/0xdc
<6> [<801038e0>] do_IRQ+0x1c/0x34
<6> [<80102110>] ret_from_irq+0x0/0x10
<6> [<801038e0>] do_IRQ+0x1c/0x34
<6> [<8d085928>] blake2s_Os_compress+0x148/0x480 [dataplane]
<6> [<8d0856ec>] capture_prep_dot11+0x9ac/0xaa0 [dataplane]
<6> [<8d005ee0>] dataplane_irq_add_decent_entropy_word+0x340/0x400 [dataplane]
<6> [<8012e67c>] __do_softirq+0xbc/0x1a4
<6> [<8d00610c>] dataplane_add_decent_entropy+0x16c/0x300 [dataplane]
<6> [<8d006060>] dataplane_add_decent_entropy+0xc0/0x300 [dataplane]
<6> [<8012ea14>] local_bh_enable+0x74/0xa8
<6> [<80430000>] setup_arch+0x26c/0x514
<6> [<8d13cc00>] mips_add_decent_entropy_slow+0x60/0xc0 [dataplane]
<6> [<8d125838>] target_add_decent_entropy_fast+0x178/0x1e0 [dataplane]
<6> [<8d001ba0>] config_enable_dataplane+0x120/0x940 [dataplane]
<6> [<8d001b98>] config_enable_dataplane+0x118/0x940 [dataplane]
<6> [<8017d0d8>] chrdev_open+0xc0/0x1c0
<6> [<8017d060>] chrdev_open+0x48/0x1c0
<6> [<8d079de0>] wireless_ingress_from_bridge+0x0/0x520 [dataplane]
<6> [<80185ecc>] path_lookup_open+0x18/0x24
<6> [<80170ac4>] do_sync_read+0xd4/0x160
<6> [<8d00c7ec>] append_to_queue_of_packets+0x8cc/0xc60 [dataplane]
<6> [<8016eb78>] __dentry_open+0xa8/0x1e4
<6> [<8016ed90>] nameidata_to_filp+0x4c/0x64
<6> [<80187ae8>] do_ioctl+0xb8/0xc8
<6> [<80187d74>] vfs_ioctl+0x27c/0x3e4
<6> [<8016e938>] get_unused_fd+0xd8/0x270
<6> [<80181e70>] getname+0x24/0x110
<6> [<80187f24>] sys_ioctl+0x48/0xb4
<6> [<8016ee98>] do_sys_open+0x8c/0xc0
<6> [<80101ad0>] xlr_ipi_handler+0xa8/0xb8
<6> [<8010ba8c>] stack_done+0x20/0x3c
<6> [<80102110>] ret_from_irq+0x0/0x10
<6> [<80237900>] memset_partial+0x44/0x6c
<6>
<6>Pretty Call Trace:
<6> SP 0x88171a80, RA 0x8010b4dc, PC 0x80109544 dump_stack+0x0/0x1c
<6>250: addiu sp,sp,-152
<6>346: frame_size 152
<6> SP 0x88171b18, RA 0x8010b4dc, PC 0x8010b4dc smp_call_function+0x1c0/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171b30, RA 0x8010b4dc, PC 0x8010b4dc smp_call_function+0x1c0/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171b48, RA 0x8010b4dc, PC 0x8010b4dc smp_call_function+0x1c0/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171b60, RA 0x8010b4dc, PC 0x8010b4dc smp_call_function+0x1c0/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171b78, RA 0x8010b4dc, PC 0x8010b4dc smp_call_function+0x1c0/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171b90, RA 0x8010b4dc, PC 0x8010b4dc smp_call_function+0x1c0/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171ba8, RA 0x8010b4dc, PC 0x8010b4dc smp_call_function+0x1c0/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171bc0, RA 0x8010b4dc, PC 0x8010b4dc smp_call_function+0x1c0/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171bd8, RA 0x8010b4dc, PC 0x8010b4dc smp_call_function+0x1c0/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171bf0, RA 0x8010b4dc, PC 0x8010b4dc smp_call_function+0x1c0/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171c08, RA 0x8010b4dc, PC 0x8010b4dc smp_call_function+0x1c0/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171c20, RA 0x8010b4dc, PC 0x8010b4dc smp_call_function+0x1c0/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171c38, RA 0x8010b4dc, PC 0x8010b4dc smp_call_function+0x1c0/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171c50, RA 0x8010b4dc, PC 0x8010b4dc smp_call_function+0x1c0/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171c68, RA 0x8010b4dc, PC 0x8010b4dc smp_call_function+0x1c0/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171c80, RA 0x8010b4dc, PC 0x8010b4dc smp_call_function+0x1c0/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171c98, RA 0x8010b4dc, PC 0x8010b4dc smp_call_function+0x1c0/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171cb0, RA 0x8010b4dc, PC 0x8010b4dc smp_call_function+0x1c0/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171cc8, RA 0x8010b4dc, PC 0x8010b4dc smp_call_function+0x1c0/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171ce0, RA 0x8010b4dc, PC 0x8010b4dc smp_call_function+0x1c0/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171cf8, RA 0x8010b4dc, PC 0x8010b4dc smp_call_function+0x1c0/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171d10, RA 0x8010b4dc, PC 0x8010b4dc smp_call_function+0x1c0/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171d28, RA 0x8010b4dc, PC 0x8010b4dc smp_call_function+0x1c0/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171d40, RA 0x8010b4dc, PC 0x8010b4dc smp_call_function+0x1c0/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6>dumping panic record into NAND1 pages 242752-262143
izeof(the_insert_locks) 4096
<6>FLOW_PREFETCH_FOR_WR_START 288, FLOW_PREFETCH_FOR_WR_END 384 (3 cachelines)
<6>NUM_MU_FW_FLOW_BLOCK 41286, num_mu_fw_flow_block 41286, MU_FW_FLOWS_PER_BLOCK 15
<6>MAX_RL_PORTS 265, RL_PORT_MASK_UNIT 32, RL_PORT_MASK_SIZE 9
<6>NUM_SNOOP_TBL_ENTRIES 16384
<6>sizeof(l3e_snoop_tbl_entry) 128 (using 102)
<6>NUM_PREFIX_ENTRIES 4096, NUM_PREFIX_BUCKETS 4096
<6>sizeof(struct ipv6_prefix_entry) = 32
<6>Voice initialized for 4096 ctl sessions
<6>Voice Ctrl Session sizes: SIP (392), SCCP (440)
<6>NAT_BITMASK_SIZE 4194304 (512 KBytes), MAX_RESERVATION_ATTEMPTS 15
<6>14336 route table entries
<6>30993 route tree nodes
<6>1024 next-hop gateways
<6>route6_tree size of btree:4
<6>route6_tree bnode32 num:1152 size :64
<6>route6_tree bnode64 num:512 size :64
<6>route6_tree lnode32 num:6336 size :128
<6>route6_tree lnode64 num:1024 size :96
<6>route6_tree rinfo num:9216 size :8
<6>route6_tree mem bnode32:73728
<6>route6_tree mem lnode32:811008
<6>route6_tree mem bnode64:32768
<6>route6_tree mem lnode64:98304
<6>route6_tree mem rinfo:73728
<6>route6_tree total mem:1089536
<6>sizeof(Neighbor_Entry) 64 (using 44)
<6>NUM_NEIGHBOR_ENTRIES 16384
<6>NUM_NEIGHBOR_TABLE_BUCKETS 4096
<6>NEIGHBOR_RETRY_INTERVAL_MSEC 4
<6>NUM_NEIGHBOR_TABLE_BUCKETS_RETRIED_PER_PASS 16
<6>arp.max_age 900000 jiffies
<6>arp.max_retries 4
<6>neighbor.max_age 900000 jiffies
<6>neighbor.max_retries 4
<6>sizeof(wlan) 1400 * MAX_WLANS 256 = 350KB
<6> sizeof(wlan_cfg) 1068
<6> sizeof(wlan_stats) 200
<6> sizeof(wlan_slow_stats) 264
<6>sizeof(ap) 28 * MAX_DATAPLANE_APS 256 = 7KB
<6> sizeof(ap_cfg) 12
<6> sizeof(ap_stats) 0
<6>sizeof(radio) 8216 * MAX_DATAPLANE_RADIOS 512 = 4108KB
<6> sizeof(radio_cfg) 1624
<6> sizeof(radio_stats) 1144
<6> sizeof(radio_slow_stats) 424
<6>sizeof(mu) 1424 * MAX_MUS 8192 = 11392KB
<6> sizeof(mu_cfg) 72
<6> sizeof(mu_stats) 568
<6> sizeof(mu_slow_stats) 320
<6>init_wireless_ioctl: Allocated 65568 bytes at 80b40000, ioctl 129(gmbi=117)
<6>NUM_[muhash]_BUCKETS 2048 w/ 8 MUs per BUCKET
<6>NUM_[probehash]_BUCKETS 1024 w/ 8 PROBE_ENTRIES per BUCKET
<6>NUM_[CS_MUHASH]_BUCKETS 16384 w/ 8 CS MUs per BUCKET
<6>NET: Registered protocol family 50
<6>evtsys: initialized dump buffer@8db22408 (1016336)
<6>MAX_MINT_VLAN_NEIGHBORS 460
<6>NUM_MINT_L2_DEFRAG_PEERS 920
<6>sizeof(mint_l2_peer) 96
<6>NUM_L2_PEER_HASH_BUCKETS 512
<6>MAX_MINT_OUTSTANDING_L2_VDEFRAGS 7360
<6>sizeof(mint_l2_vdefrag) 240
<6>NUM_L2SHIM_REAS_ID_BUCKETS 256
<6>sizeof(mint_hdr) 48, sizeof(l2_mint_shim) 4 sizeof(mint_vlan_hdr2) 6
<6>MINT_PKT_RESERVE 108, MINT_PKT_RESERVE_NO_BASIL 76, MINT_PKT_NEXT_FRAG_RESERVE 52
<6>MINT_PAYLOAD_FIRST_FRAG 1392, MINT_PAYLOAD_FIRST_FRAG_NO_BASIL 1424, MINT_PAYLOAD_NEXT_FRAG 1448
<6>mint.switch_addr = 70.38.0A.F9
<6>MAX_MINT_NEIGHBORS 1843
<6>NUM_NEIGHBOR_ENTRIES 2048, NUM_NEIGHBOR_BUCKETS 256
<6>sizeof(Mint_Neighbor) 48
<6>MINT_NEIGHBOR_FILTER_SIZE 8192
<6>MAX_MINT_ROUTES 7372
<6>max_mint_routes 7372
<6>NUM_ROUTE_ENTRIES 8192, NUM_ROUTE_BUCKETS 1024
<6>num_route_entries 8192, num_route_buckets 1024
<6>sizeof(Mint_Route) 32
<6>MAX_MINT_HELLOS 1843
<6>NUM_HELLO_ENTRIES 2048, NUM_HELLO_BUCKETS 256
<6>sizeof(Mint_Hello) 32
<6>NUM_HOST_ENTRIES 8192, NUM_HOST_BUCKETS 1024
<6>num_host_entries 8192, num_host_buckets 1024
<6>sizeof(Mint_Host) 8
<6>sizeof(BasilSC) 184
<6>sizeof(BasilKey) 88
<6>NUM_L2TPv3_ENTRIES 1023, MAX_L2TPv3_MANUAL_SESSIONS 63
<6>PBR_MAX_NEXT_HOPS_PER_ROUTE_MAP=2, PBR_MAX_DEF_NEXT_HOPS_PER_ROUTE_MAP=1, PBR_MAX_ROUTE_MAP_NODES=64, PBR_MAX_ROUTE_MAPS=32
<6>is_access_point:0, is_switch:1
<6>sizeof(the_config) = 6036432
<6>offsetof(debug_level) = 0
<6>offsetof(switch_mac) = 16
<6>offsetof(router) = 196
<6>offsetof(host) = 228
<6>offsetof(l3acl) = 308
<6>offsetof(fw) = 312
<6>offsetof(dot11) = 11784
<6>offsetof(stp_forwarding_ports) = 298218
<6>offsetof(trunks) = 298250
<6>offsetof(ext_ports) = 298264
<6>offsetof(ports) = 298504
<6>offsetof(vlans) = 298648
<6>offsetof(interfaces) = 2641560
<6>sizeof(Port_Config) = 16
<6>sizeof(External_Port_Config) = 36
<6>sizeof(STP_Port_Config) = 16
<6>sizeof(Vlan_Config) = 572
<6>sizeof(Interface_Config) = 764
<6>sizeof(Lacp_Config) = 7
<6>sizeof(Firewall_Config) = 11468
<6>sizeof(DataplaneThread) = 23456, the_dataplanethreads 8d77e820
<6>offsetof(profile) = 36
<6>offsetof(stats) = 7040
<6>offsetof(tx_queues) = 7072
<6>offsetof(crypto_queues) = 9632
<6>offsetof(msgs) = 9700
<6>offsetof(print) = 13144
<6>MAX_NORMAL_PKTS_QUEUED_FOR_TX = 64, MAX_FLOODING_PKTS_QUEUED_FOR_TX = 91
<6>dataplane_thread_stacks at eed2d1e0, size DATAPLANE_THREAD_STACK_SIZE 9216 *

If anyone can interprate this ? of offer any advice

RE: RFS7000 Crashing - Wing 5.8.5

Daniel_Fraile_V — Wed, 20 Sep 2017 16:53:00 GMT

Hello Phil,

It seems as you said, a soft lockup, caused by the dataplane process. Could you please give us a better view of how are those APs deployed? Different domains and level 2 MiNT links? Can you please share the output of the following commands on the panicking device?

#show mint links
#service show top
#show cluster members detail

How often is that panic happening?

Thanks in advance.

RE: RFS7000 Crashing - Wing 5.8.5

Phil_storey — Wed, 20 Sep 2017 17:31:00 GMT

link ip-172.17.146.105:24576 at level 1, 1 adjacencies
link ip-172.17.146.106:24576 at level 1, 0 adjacencies
link vlan-1 at level 1, 15 adjacencies, DIS 70.38.0A.F9 (self)

PID STATUS RSS PPID %CPU %MEM COMMAND
1061 R 334641052 1.9 15.0 cfgd
1132 S 331441122 0.1 14.8 ssm
1060 S 9680 1042 0.0 4.3 rim
12854 S 8860 12842 0.0 3.9 mapsh
1167 S 7156 1157 0.0 3.2 hsd
1187 S 6984 1179 0.0 3.1 dpd2
1449 S 4736 1441 0.0 2.1 snmpd
1514 S 3640 1492 0.0 1.6 lighttpd
1104 S 3140 1096 4.3 1.4 nsm
1029 S 2756 1021 0.0 1.2 logd

---------------------------------------------------------------------------------------------------------
ID MAC MODE AP COUNT AAP COUNT AP LICENSE AAP LICENSE VERSION
---------------------------------------------------------------------------------------------------------
70.38.0A.F9 00-15-70-38-0A-F9 Standby 0 0 0 0 5.8.5.0-016R
70.81.BE.8E 00-15-70-81-BE-8E Active 0 15 128 0 5.8.5.0-016R

RE: RFS7000 Crashing - Wing 5.8.5

Daniel_Fraile_V — Wed, 20 Sep 2017 17:57:00 GMT

Hi Phil,

Is the device 70.38.0A.F9 acting as well as cluster master? You can check it with the following command:

#show cluster members

RE: RFS7000 Crashing - Wing 5.8.5

Phil_storey — Wed, 20 Sep 2017 19:10:00 GMT

Hi
No, 70:38:0A:F0 is in standby

RE: RFS7000 Crashing - Wing 5.8.5

Daniel_Fraile_V — Wed, 20 Sep 2017 19:10:00 GMT

Hi again Phil,

Sorry for the confusion. What I meant is, if the controller with MiNT ID 70:38:0A:F0 has the master status on the cluster showing up as true. Can you please share the output of the following command?

#show cluster members

RE: RFS7000 Crashing - Wing 5.8.5

Phil_storey — Wed, 20 Sep 2017 19:10:00 GMT

rfs7000-Backup 70.38.0A.F9 00-15-70-38-0A-F9 False standby 00:01:01 ago
rfs7000-Primary 70.81.BE.8E 00-15-70-81-BE-8E True active self

RE: RFS7000 Crashing - Wing 5.8.5

Phil_storey — Wed, 20 Sep 2017 19:10:00 GMT

rfs7000-Backup 70.38.0A.F9 00-15-70-38-0A-F9 False standby 00:01:01 ago
rfs7000-Primary 70.81.BE.8E 00-15-70-81-BE-8E True active self

RE: RFS7000 Crashing - Wing 5.8.5

Alona — Fri, 22 Sep 2017 08:19:00 GMT

I don't think your crashes are related to your problems and besides the panic which was seen during taking of packet capture most porbably - the rest are not crashes or you are just not putting crash locations. And the logs above are from 6521 - not the Aps you have problems with.
You need to have GTAC look at your config and check if config is correct.

RE: RFS7000 Crashing - Wing 5.8.5

Phil_storey — Fri, 22 Sep 2017 11:04:00 GMT

Hi
I did no packet capture, the painc wa in the crash file location on the RFS,
What are these messages that are repeated ?
<6>Pretty Call Trace:
<6> SP 0x88171a80, RA 0x8010b4dc, PC 0x80109544 dump_stack+0x0/0x1c
<6>250: addiu sp,sp,-152
<6>346: frame_size 152
<6> SP 0x88171b18, RA 0x8010b4dc, PC 0x8010b4dc smp_call_function+0x1c0/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171b30, RA 0x8010b4dc, PC 0x8010b4dc smp_call_function+0x1c0/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171b48, RA 0x8010b4dc, PC 0x8010b4dc smp_call_function+0x1c0/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171b60, RA 0x8010b4dc, PC 0x8010b4dc smp_call_function+0x1c0/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171b78, RA 0x8010b4dc, PC 0x8010b4dc smp_call_function+0x1c0/0x20c

RE: RFS7000 Crashing - Wing 5.8.5

Alona — Fri, 22 Sep 2017 22:01:00 GMT

This is just stack fuction call. Is this panic hapenning constantly - or you just have one time panic occurance? You can look at the timestamp and see how long ago panic occured. Again - you said MUs are not connecting and unless your controller is constantly rebooting - that panic is not what's causing you a problem. Probably something else in the config.

topic RE: RFS7000 Crashing - Wing 5.8.5 in ExtremeWireless (WiNG)

RFS7000 Crashing - Wing 5.8.5

RE: RFS7000 Crashing - Wing 5.8.5

RE: RFS7000 Crashing - Wing 5.8.5

RE: RFS7000 Crashing - Wing 5.8.5

RE: RFS7000 Crashing - Wing 5.8.5

RE: RFS7000 Crashing - Wing 5.8.5

RE: RFS7000 Crashing - Wing 5.8.5

RE: RFS7000 Crashing - Wing 5.8.5

RE: RFS7000 Crashing - Wing 5.8.5

RE: RFS7000 Crashing - Wing 5.8.5

RE: RFS7000 Crashing - Wing 5.8.5