RFS7000 Crashing - Wing 5.8.5

  • 0
  • 2
  • Question
  • Updated 1 year ago
  • Answered
Hi
  over the last couple of day the RFS7K has dropped off the network and stops all MU form connecting. The WIF is still visable ( Ap's are 7131 & 7532 ) but no connection can be made.
I have looked on the Primary unit and there was some crash files about 6 in total but dating from earlier in the this month.

But its all a bit meaningless to me:

Thu Sep  7 10:39:35 2017: /home/spin/workspace/ncap_br2/build_br2/obj/qs5/src/sys/cfgd/remote_debug/server.py:964 server_tasklet() session args = {'xfer_args': {'tzsp_encap': True}, 'output_type': 9}
Return success for start_session 

Thu Sep  7 10:39:35 2017: /home/spin/workspace/ncap_br2/build_br2/obj/qs5/src/sys/cfgd/remote_debug/server.py:1034 session_initiator_tasklet() requesting host 70.81.BE.8E
Thu Sep  7 10:39:35 2017: /home/spin/workspace/ncap_br2/build_br2/obj/qs5/src/sys/cfgd/remote_debug/client.py:778 req_listen_tasklet() Peer 70.81.BE.8E connected over seqpacket port 36

Thu Sep  7 10:39:35 2017: /home/spin/workspace/ncap_br2/build_br2/obj/qs5/src/sys/cfgd/remote_debug/server.py:1066 session_initiator_tasklet() notified 1 of 1 hosts

In handle_request_tasklet =====> req message <cfgd.remote_debug.common.RequestPacket object at 0x2c68789c>
Thu Sep  7 10:39:35 2017: /home/spin/workspace/ncap_br2/build_br2/obj/qs5/src/sys/cfgd/remote_debug/client.py:734 handle_request_tasklet() Got request 3
Thu Sep  7 10:39:35 2017: /home/spin/workspace/ncap_br2/build_br2/obj/qs5/src/sys/cfgd/remote_debug/client.py:182 handle_live_pktcap_request() handle_live_pktcap_request from (70.81.BE.8E, 54038)

How ever when I looked on the backup unit there were loads more like this

Thu Sep  7 10:45:28 2017: /home/spin/workspace/ncap_br2/build_br2/obj/qs5/src/sys/cfgd/idpd/role.py:193 update_acl_to_dp() [role def Guest] ACTION: Update out acl for DP
Thu Sep  7 10:45:28 2017: /home/spin/workspace/ncap_br2/build_br2/obj/qs5/src/sys/cfgd/idpd/role.py:237 update_acl_to_dp() [role def Guest] No ip out acl configured, applying default
Thu Sep  7 10:45:28 2017: /home/spin/workspace/ncap_br2/build_br2/obj/qs5/src/sys/cfgd/idpd/role.py:239 update_acl_to_dp() [role def Guest] Updating DEFAULT acl to DP
Thu Sep  7 10:45:28 2017: /home/spin/workspace/ncap_br2/build_br2/obj/qs5/src/sys/cfgd/idpd/role.py:237 update_acl_to_dp() [role def Guest] No ipv6 out acl configured, applying default
Thu Sep  7 10:45:28 2017: /home/spin/workspace/ncap_br2/build_br2/obj/qs5/src/sys/cfgd/idpd/role.py:239 update_acl_to_dp() [role def Guest] Updating DEFAULT acl to DP

There was also a panic file dated today- it seems to be complaining about a soft lockup

NUM_DATA_THREADS = 258048
<6>highmem remaining: 19023KB/288MB (6%)
<6>module_alloc:58 (size 169360)
<6>module_alloc:58 (size 128)
<6>Dataplane Hardware Services Layer for IP Infusion
<6>NET: Registered protocol family 48
<6>NET: Registered protocol family 40
<6>NET: Registered protocol family 42
<6>module_alloc:58 (size 26240)
<6>Compact Flash Driver Module Installed
<6>ge1 { no link }
<6>me1: DSPCFG accepted after 0 usec.
<6>ADDRCONF(NETDEV_UP): me1: link is not ready
<6>ge1 { autoneg'ed 1Gb/s, autoneg'ed full duplex, Cu }
<6>me1: DSPCFG accepted after 0 usec.
<6>ge1 { no link }
<6>ge1 { autoneg'ed 1Gb/s, autoneg'ed full duplex, Cu }
<6>ge1 { no link }
<3>BUG: soft lockup detected on CPU#0!
<6>Cpu 0
<6>$ 0   : 00000000 50104c00 00000008 0000000c
<6>$ 4   : 00000000 88171cd8 88171c98 00000000
<6>$ 8   : 510e523f e07c2654 8d140000 3ebb3f3e
<6>$12   : 8dbde1b6 00000040 88171cd0 88171cd4
<6>$16   : 8d75c1f4 8d75c1f4 80923974 88171cd8
<6>$20   : 88171c98 00000000 8d18fe30 00000000
<6>$24   : 0d1561b6 88171cbc                 
<6>$28   : 88170000 88171c78 000dc690 8d085928
<6>Hi    : 03079680
<6>Lo    : 02a6a3b0
<6>epc   : 8d0856ec capture_prep_dot11+0x9ac/0xaa0 [dataplane]     Tainted: PF   
<6>ra    : 8d085928 blake2s_Os_compress+0x148/0x480 [dataplane]
<6>Status: 50104c03    KERNEL EXL IE
<6>Cause : 00808000
<6>PrId  : 000c0904
<0>Kernel panic - not syncing: softlockup: hung tasks
<6> Badness in smp_call_function at arch/mips/kernel/smp.c:159
<6>Raw Call Trace:
<6> [<80127534>] panic+0x74/0x1c0
<6> [<80127514>] panic+0x54/0x1c0
<6> [<8014c200>] watchdog+0x0/0x80
<6> [<80430000>] setup_arch+0x26c/0x514
<6> [<801081f0>] timer_interrupt+0xa0/0x248
<6> [<80430000>] setup_arch+0x26c/0x514
<6> [<8014c32c>] handle_IRQ_event+0x74/0xdc
<6> [<8014c4e8>] __do_IRQ+0x154/0x1fc
<6> [<8014c32c>] handle_IRQ_event+0x74/0xdc
<6> [<801038e0>] do_IRQ+0x1c/0x34
<6> [<80102110>] ret_from_irq+0x0/0x10
<6> [<801038e0>] do_IRQ+0x1c/0x34
<6> [<8d085928>] blake2s_Os_compress+0x148/0x480 [dataplane]
<6> [<8d0856ec>] capture_prep_dot11+0x9ac/0xaa0 [dataplane]
<6> [<8d005ee0>] dataplane_irq_add_decent_entropy_word+0x340/0x400 [dataplane]
<6> [<8012e67c>] __do_softirq+0xbc/0x1a4
<6> [<8d00610c>] dataplane_add_decent_entropy+0x16c/0x300 [dataplane]
<6> [<8d006060>] dataplane_add_decent_entropy+0xc0/0x300 [dataplane]
<6> [<8012ea14>] local_bh_enable+0x74/0xa8
<6> [<80430000>] setup_arch+0x26c/0x514
<6> [<8d13cc00>] mips_add_decent_entropy_slow+0x60/0xc0 [dataplane]
<6> [<8d125838>] target_add_decent_entropy_fast+0x178/0x1e0 [dataplane]
<6> [<8d001ba0>] config_enable_dataplane+0x120/0x940 [dataplane]
<6> [<8d001b98>] config_enable_dataplane+0x118/0x940 [dataplane]
<6> [<8017d0d8>] chrdev_open+0xc0/0x1c0
<6> [<8017d060>] chrdev_open+0x48/0x1c0
<6> [<8d079de0>] wireless_ingress_from_bridge+0x0/0x520 [dataplane]
<6> [<80185ecc>] path_lookup_open+0x18/0x24
<6> [<80170ac4>] do_sync_read+0xd4/0x160
<6> [<8d00c7ec>] append_to_queue_of_packets+0x8cc/0xc60 [dataplane]
<6> [<8016eb78>] __dentry_open+0xa8/0x1e4
<6> [<8016ed90>] nameidata_to_filp+0x4c/0x64
<6> [<80187ae8>] do_ioctl+0xb8/0xc8
<6> [<80187d74>] vfs_ioctl+0x27c/0x3e4
<6> [<8016e938>] get_unused_fd+0xd8/0x270
<6> [<80181e70>] getname+0x24/0x110
<6> [<80187f24>] sys_ioctl+0x48/0xb4
<6> [<8016ee98>] do_sys_open+0x8c/0xc0
<6> [<80101ad0>] xlr_ipi_handler+0xa8/0xb8
<6> [<8010ba8c>] stack_done+0x20/0x3c
<6> [<80102110>] ret_from_irq+0x0/0x10
<6> [<80237900>] memset_partial+0x44/0x6c
<6>
<6>Pretty Call Trace:
<6> SP 0x88171a80, RA 0x8010b514, PC 0x80109544 dump_stack+0x0/0x1c
<6>250: addiu sp,sp,-152
<6>346: frame_size 152
<6> SP 0x88171b18, RA 0x8010b514, PC 0x8010b514 smp_call_function+0x1f8/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171b30, RA 0x8010b514, PC 0x8010b514 smp_call_function+0x1f8/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171b48, RA 0x8010b514, PC 0x8010b514 smp_call_function+0x1f8/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171b60, RA 0x8010b514, PC 0x8010b514 smp_call_function+0x1f8/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171b78, RA 0x8010b514, PC 0x8010b514 smp_call_function+0x1f8/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171b90, RA 0x8010b514, PC 0x8010b514 smp_call_function+0x1f8/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171ba8, RA 0x8010b514, PC 0x8010b514 smp_call_function+0x1f8/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171bc0, RA 0x8010b514, PC 0x8010b514 smp_call_function+0x1f8/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171bd8, RA 0x8010b514, PC 0x8010b514 smp_call_function+0x1f8/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171bf0, RA 0x8010b514, PC 0x8010b514 smp_call_function+0x1f8/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171c08, RA 0x8010b514, PC 0x8010b514 smp_call_function+0x1f8/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171c20, RA 0x8010b514, PC 0x8010b514 smp_call_function+0x1f8/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171c38, RA 0x8010b514, PC 0x8010b514 smp_call_function+0x1f8/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171c50, RA 0x8010b514, PC 0x8010b514 smp_call_function+0x1f8/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171c68, RA 0x8010b514, PC 0x8010b514 smp_call_function+0x1f8/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171c80, RA 0x8010b514, PC 0x8010b514 smp_call_function+0x1f8/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171c98, RA 0x8010b514, PC 0x8010b514 smp_call_function+0x1f8/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171cb0, RA 0x8010b514, PC 0x8010b514 smp_call_function+0x1f8/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171cc8, RA 0x8010b514, PC 0x8010b514 smp_call_function+0x1f8/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171ce0, RA 0x8010b514, PC 0x8010b514 smp_call_function+0x1f8/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171cf8, RA 0x8010b514, PC 0x8010b514 smp_call_function+0x1f8/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171d10, RA 0x8010b514, PC 0x8010b514 smp_call_function+0x1f8/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171d28, RA 0x8010b514, PC 0x8010b514 smp_call_function+0x1f8/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171d40, RA 0x8010b514, PC 0x8010b514 smp_call_function+0x1f8/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6>Badness in smp_call_function at arch/mips/kernel/smp.c:160
<6>Raw Call Trace:
<6> [<80127534>] panic+0x74/0x1c0
<6> [<80127514>] panic+0x54/0x1c0
<6> [<8014c200>] watchdog+0x0/0x80
<6> [<80430000>] setup_arch+0x26c/0x514
<6> [<801081f0>] timer_interrupt+0xa0/0x248
<6> [<80430000>] setup_arch+0x26c/0x514
<6> [<8014c32c>] handle_IRQ_event+0x74/0xdc
<6> [<8014c4e8>] __do_IRQ+0x154/0x1fc
<6> [<8014c32c>] handle_IRQ_event+0x74/0xdc
<6> [<801038e0>] do_IRQ+0x1c/0x34
<6> [<80102110>] ret_from_irq+0x0/0x10
<6> [<801038e0>] do_IRQ+0x1c/0x34
<6> [<8d085928>] blake2s_Os_compress+0x148/0x480 [dataplane]
<6> [<8d0856ec>] capture_prep_dot11+0x9ac/0xaa0 [dataplane]
<6> [<8d005ee0>] dataplane_irq_add_decent_entropy_word+0x340/0x400 [dataplane]
<6> [<8012e67c>] __do_softirq+0xbc/0x1a4
<6> [<8d00610c>] dataplane_add_decent_entropy+0x16c/0x300 [dataplane]
<6> [<8d006060>] dataplane_add_decent_entropy+0xc0/0x300 [dataplane]
<6> [<8012ea14>] local_bh_enable+0x74/0xa8
<6> [<80430000>] setup_arch+0x26c/0x514
<6> [<8d13cc00>] mips_add_decent_entropy_slow+0x60/0xc0 [dataplane]
<6> [<8d125838>] target_add_decent_entropy_fast+0x178/0x1e0 [dataplane]
<6> [<8d001ba0>] config_enable_dataplane+0x120/0x940 [dataplane]
<6> [<8d001b98>] config_enable_dataplane+0x118/0x940 [dataplane]
<6> [<8017d0d8>] chrdev_open+0xc0/0x1c0
<6> [<8017d060>] chrdev_open+0x48/0x1c0
<6> [<8d079de0>] wireless_ingress_from_bridge+0x0/0x520 [dataplane]
<6> [<80185ecc>] path_lookup_open+0x18/0x24
<6> [<80170ac4>] do_sync_read+0xd4/0x160
<6> [<8d00c7ec>] append_to_queue_of_packets+0x8cc/0xc60 [dataplane]
<6> [<8016eb78>] __dentry_open+0xa8/0x1e4
<6> [<8016ed90>] nameidata_to_filp+0x4c/0x64
<6> [<80187ae8>] do_ioctl+0xb8/0xc8
<6> [<80187d74>] vfs_ioctl+0x27c/0x3e4
<6> [<8016e938>] get_unused_fd+0xd8/0x270
<6> [<80181e70>] getname+0x24/0x110
<6> [<80187f24>] sys_ioctl+0x48/0xb4
<6> [<8016ee98>] do_sys_open+0x8c/0xc0
<6> [<80101ad0>] xlr_ipi_handler+0xa8/0xb8
<6> [<8010ba8c>] stack_done+0x20/0x3c
<6> [<80102110>] ret_from_irq+0x0/0x10
<6> [<80237900>] memset_partial+0x44/0x6c
<6>
<6>Pretty Call Trace:
<6> SP 0x88171a80, RA 0x8010b4dc, PC 0x80109544 dump_stack+0x0/0x1c
<6>250: addiu sp,sp,-152
<6>346: frame_size 152
<6> SP 0x88171b18, RA 0x8010b4dc, PC 0x8010b4dc smp_call_function+0x1c0/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171b30, RA 0x8010b4dc, PC 0x8010b4dc smp_call_function+0x1c0/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171b48, RA 0x8010b4dc, PC 0x8010b4dc smp_call_function+0x1c0/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171b60, RA 0x8010b4dc, PC 0x8010b4dc smp_call_function+0x1c0/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171b78, RA 0x8010b4dc, PC 0x8010b4dc smp_call_function+0x1c0/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171b90, RA 0x8010b4dc, PC 0x8010b4dc smp_call_function+0x1c0/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171ba8, RA 0x8010b4dc, PC 0x8010b4dc smp_call_function+0x1c0/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171bc0, RA 0x8010b4dc, PC 0x8010b4dc smp_call_function+0x1c0/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171bd8, RA 0x8010b4dc, PC 0x8010b4dc smp_call_function+0x1c0/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171bf0, RA 0x8010b4dc, PC 0x8010b4dc smp_call_function+0x1c0/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171c08, RA 0x8010b4dc, PC 0x8010b4dc smp_call_function+0x1c0/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171c20, RA 0x8010b4dc, PC 0x8010b4dc smp_call_function+0x1c0/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171c38, RA 0x8010b4dc, PC 0x8010b4dc smp_call_function+0x1c0/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171c50, RA 0x8010b4dc, PC 0x8010b4dc smp_call_function+0x1c0/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171c68, RA 0x8010b4dc, PC 0x8010b4dc smp_call_function+0x1c0/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171c80, RA 0x8010b4dc, PC 0x8010b4dc smp_call_function+0x1c0/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171c98, RA 0x8010b4dc, PC 0x8010b4dc smp_call_function+0x1c0/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171cb0, RA 0x8010b4dc, PC 0x8010b4dc smp_call_function+0x1c0/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171cc8, RA 0x8010b4dc, PC 0x8010b4dc smp_call_function+0x1c0/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171ce0, RA 0x8010b4dc, PC 0x8010b4dc smp_call_function+0x1c0/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171cf8, RA 0x8010b4dc, PC 0x8010b4dc smp_call_function+0x1c0/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171d10, RA 0x8010b4dc, PC 0x8010b4dc smp_call_function+0x1c0/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171d28, RA 0x8010b4dc, PC 0x8010b4dc smp_call_function+0x1c0/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171d40, RA 0x8010b4dc, PC 0x8010b4dc smp_call_function+0x1c0/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6>dumping panic record into NAND1 pages 242752-262143
izeof(the_insert_locks) 4096
<6>FLOW_PREFETCH_FOR_WR_START 288, FLOW_PREFETCH_FOR_WR_END 384 (3 cachelines)
<6>NUM_MU_FW_FLOW_BLOCK 41286, num_mu_fw_flow_block 41286, MU_FW_FLOWS_PER_BLOCK 15
<6>MAX_RL_PORTS 265, RL_PORT_MASK_UNIT 32, RL_PORT_MASK_SIZE 9
<6>NUM_SNOOP_TBL_ENTRIES 16384
<6>sizeof(l3e_snoop_tbl_entry) 128 (using 102)
<6>NUM_PREFIX_ENTRIES 4096, NUM_PREFIX_BUCKETS 4096
<6>sizeof(struct ipv6_prefix_entry) = 32
<6>Voice initialized for 4096 ctl sessions
<6>Voice Ctrl Session sizes: SIP (392), SCCP (440)
<6>NAT_BITMASK_SIZE 4194304 (512 KBytes), MAX_RESERVATION_ATTEMPTS 15
<6>14336 route table entries
<6>30993 route tree nodes
<6>1024 next-hop gateways
<6>route6_tree size of btree:4
<6>route6_tree bnode32 num:1152 size :64
<6>route6_tree bnode64 num:512 size :64
<6>route6_tree lnode32 num:6336 size :128
<6>route6_tree lnode64 num:1024 size :96
<6>route6_tree rinfo  num:9216 size :8
<6>route6_tree mem bnode32:73728
<6>route6_tree mem lnode32:811008
<6>route6_tree mem bnode64:32768
<6>route6_tree mem lnode64:98304
<6>route6_tree mem rinfo:73728
<6>route6_tree total mem:1089536
<6>sizeof(Neighbor_Entry) 64 (using 44)
<6>NUM_NEIGHBOR_ENTRIES 16384
<6>NUM_NEIGHBOR_TABLE_BUCKETS 4096
<6>NEIGHBOR_RETRY_INTERVAL_MSEC 4
<6>NUM_NEIGHBOR_TABLE_BUCKETS_RETRIED_PER_PASS 16
<6>arp.max_age 900000 jiffies
<6>arp.max_retries 4
<6>neighbor.max_age 900000 jiffies
<6>neighbor.max_retries 4
<6>sizeof(wlan) 1400 * MAX_WLANS 256 = 350KB
<6>  sizeof(wlan_cfg) 1068
<6>  sizeof(wlan_stats) 200
<6>  sizeof(wlan_slow_stats) 264
<6>sizeof(ap) 28 * MAX_DATAPLANE_APS 256 = 7KB
<6>  sizeof(ap_cfg) 12
<6>  sizeof(ap_stats) 0
<6>sizeof(radio) 8216 * MAX_DATAPLANE_RADIOS 512 = 4108KB
<6>  sizeof(radio_cfg) 1624
<6>  sizeof(radio_stats) 1144
<6>  sizeof(radio_slow_stats) 424
<6>sizeof(mu) 1424 * MAX_MUS 8192 = 11392KB
<6>  sizeof(mu_cfg) 72
<6>  sizeof(mu_stats) 568
<6>  sizeof(mu_slow_stats) 320
<6>init_wireless_ioctl: Allocated 65568 bytes at 80b40000, ioctl 129(gmbi=117)
<6>NUM_[MUHASH]_BUCKETS 2048 w/ 8 MUs per BUCKET
<6>NUM_[PROBEHASH]_BUCKETS 1024 w/ 8 PROBE_ENTRIES per BUCKET
<6>NUM_[CS_MUHASH]_BUCKETS 16384 w/ 8 CS MUs per BUCKET
<6>NET: Registered protocol family 50
<6>evtsys: initialized dump buffer @8db22408 (1016336)
<6>MAX_MINT_VLAN_NEIGHBORS 460
<6>NUM_MINT_L2_DEFRAG_PEERS 920
<6>sizeof(mint_l2_peer) 96
<6>NUM_L2_PEER_HASH_BUCKETS 512
<6>MAX_MINT_OUTSTANDING_L2_VDEFRAGS 7360
<6>sizeof(mint_l2_vdefrag) 240
<6>NUM_L2SHIM_REAS_ID_BUCKETS 256
<6>sizeof(mint_hdr) 48, sizeof(l2_mint_shim) 4 sizeof(mint_vlan_hdr2) 6
<6>MINT_PKT_RESERVE 108, MINT_PKT_RESERVE_NO_BASIL 76, MINT_PKT_NEXT_FRAG_RESERVE 52
<6>MINT_PAYLOAD_FIRST_FRAG 1392, MINT_PAYLOAD_FIRST_FRAG_NO_BASIL 1424, MINT_PAYLOAD_NEXT_FRAG 1448
<6>mint.switch_addr = 70.38.0A.F9
<6>MAX_MINT_NEIGHBORS 1843
<6>NUM_NEIGHBOR_ENTRIES 2048, NUM_NEIGHBOR_BUCKETS 256
<6>sizeof(Mint_Neighbor) 48
<6>MINT_NEIGHBOR_FILTER_SIZE 8192
<6>MAX_MINT_ROUTES 7372
<6>max_mint_routes 7372
<6>NUM_ROUTE_ENTRIES 8192, NUM_ROUTE_BUCKETS 1024
<6>num_route_entries 8192, num_route_buckets 1024
<6>sizeof(Mint_Route) 32
<6>MAX_MINT_HELLOS 1843
<6>NUM_HELLO_ENTRIES 2048, NUM_HELLO_BUCKETS 256
<6>sizeof(Mint_Hello) 32
<6>NUM_HOST_ENTRIES 8192, NUM_HOST_BUCKETS 1024
<6>num_host_entries 8192, num_host_buckets 1024
<6>sizeof(Mint_Host) 8
<6>sizeof(BasilSC) 184
<6>sizeof(BasilKey) 88
<6>NUM_L2TPv3_ENTRIES 1023, MAX_L2TPv3_MANUAL_SESSIONS 63
<6>PBR_MAX_NEXT_HOPS_PER_ROUTE_MAP=2, PBR_MAX_DEF_NEXT_HOPS_PER_ROUTE_MAP=1, PBR_MAX_ROUTE_MAP_NODES=64, PBR_MAX_ROUTE_MAPS=32
<6>is_access_point:0, is_switch:1
<6>sizeof(the_config) = 6036432
<6>offsetof(debug_level) = 0
<6>offsetof(switch_mac) = 16
<6>offsetof(router) = 196
<6>offsetof(host) = 228
<6>offsetof(l3acl) = 308
<6>offsetof(fw) = 312
<6>offsetof(dot11) = 11784
<6>offsetof(stp_forwarding_ports) = 298218
<6>offsetof(trunks) = 298250
<6>offsetof(ext_ports) = 298264
<6>offsetof(ports) = 298504
<6>offsetof(vlans) = 298648
<6>offsetof(interfaces) = 2641560
<6>sizeof(Port_Config) = 16
<6>sizeof(External_Port_Config) = 36
<6>sizeof(STP_Port_Config) = 16
<6>sizeof(Vlan_Config) = 572
<6>sizeof(Interface_Config) = 764
<6>sizeof(Lacp_Config) = 7
<6>sizeof(Firewall_Config) = 11468
<6>sizeof(DataplaneThread) = 23456, the_dataplanethreads 8d77e820
<6>offsetof(profile) = 36
<6>offsetof(stats) = 7040
<6>offsetof(tx_queues) = 7072
<6>offsetof(crypto_queues) = 9632
<6>offsetof(msgs) = 9700
<6>offsetof(print) = 13144
<6>MAX_NORMAL_PKTS_QUEUED_FOR_TX = 64, MAX_FLOODING_PKTS_QUEUED_FOR_TX = 91
<6>dataplane_thread_stacks at eed2d1e0, size DATAPLANE_THREAD_STACK_SIZE 9216 *

If anyone can interprate this ? of offer any advice
Photo of Phil storey

Phil storey

  • 1,254 Points 1k badge 2x thumb

Posted 1 year ago

  • 0
  • 2
Photo of Daniel Fraile Villalba

Daniel Fraile Villalba, Escalation Support Engineer

  • 790 Points 500 badge 2x thumb
Hello Phil,

It seems as you said, a soft lockup, caused by the dataplane process. Could you please give us a better view of how are those APs deployed? Different domains and level 2 MiNT links? Can you please share the output of the following commands on the panicking device?

#show mint links
#service show top
#show cluster members detail

How often is that panic happening?  

Thanks in advance.
Photo of Phil storey

Phil storey

  • 1,254 Points 1k badge 2x thumb
link ip-172.17.146.105:24576 at level 1, 1 adjacencies
link ip-172.17.146.106:24576 at level 1, 0 adjacencies
link vlan-1 at level 1, 15 adjacencies, DIS 70.38.0A.F9 (self)


PID  STATUS   RSS  PPID %CPU %MEM COMMAND
 1061  R        334641052  1.9 15.0 cfgd
 1132  S        331441122  0.1 14.8 ssm
 1060  S        9680 1042  0.0  4.3 rim
 12854 S        8860 12842 0.0  3.9 mapsh
 1167  S        7156 1157  0.0  3.2 hsd
 1187  S        6984 1179  0.0  3.1 dpd2
 1449  S        4736 1441  0.0  2.1 snmpd
 1514  S        3640 1492  0.0  1.6 lighttpd
 1104  S        3140 1096  4.3  1.4 nsm
 1029  S        2756 1021  0.0  1.2 logd

---------------------------------------------------------------------------------------------------------
       ID              MAC            MODE   AP COUNT AAP COUNT AP LICENSE AAP LICENSE      VERSION
---------------------------------------------------------------------------------------------------------
  70.38.0A.F9   00-15-70-38-0A-F9   Standby   0        0         0          0           5.8.5.0-016R
  70.81.BE.8E   00-15-70-81-BE-8E   Active    0        15        128        0           5.8.5.0-016R
Photo of Daniel Fraile Villalba

Daniel Fraile Villalba, Escalation Support Engineer

  • 790 Points 500 badge 2x thumb
Hi Phil,

Is the device 70.38.0A.F9 acting as well as cluster master? You can check it with the following command:

#show cluster members  
Photo of Phil storey

Phil storey

  • 1,254 Points 1k badge 2x thumb
Hi
  No, 70:38:0A:F0 is in standby
Photo of Daniel Fraile Villalba

Daniel Fraile Villalba, Escalation Support Engineer

  • 790 Points 500 badge 2x thumb
Hi again Phil,

Sorry for the confusion. What I meant is, if the controller with MiNT ID 70:38:0A:F0 has the master status on the cluster showing up as true. Can you please share the output of the following command?

#show cluster members  
Photo of Phil storey

Phil storey

  • 1,254 Points 1k badge 2x thumb
  rfs7000-Backup    70.38.0A.F9   00-15-70-38-0A-F9   False   standby           00:01:01 ago
  rfs7000-Primary   70.81.BE.8E   00-15-70-81-BE-8E   True    active            self
Photo of Phil storey

Phil storey

  • 1,254 Points 1k badge 2x thumb
  rfs7000-Backup    70.38.0A.F9   00-15-70-38-0A-F9   False   standby           00:01:01 ago
  rfs7000-Primary   70.81.BE.8E   00-15-70-81-BE-8E   True    active            self
Photo of Alona

Alona, Employee

  • 1,700 Points 1k badge 2x thumb
I don't think your crashes are related to your problems and besides the panic which was seen during taking of packet capture most porbably - the rest are not crashes or you are just not putting crash locations. And the logs above are from 6521 - not the Aps you have problems with.
You  need to have GTAC look at your config and check if config is correct.
Photo of Phil storey

Phil storey

  • 1,254 Points 1k badge 2x thumb
Hi
 I did no packet capture, the painc wa in the crash file location on the RFS,
What are these messages that are repeated ?
<6>Pretty Call Trace:
<6> SP 0x88171a80, RA 0x8010b4dc, PC 0x80109544 dump_stack+0x0/0x1c
<6>250: addiu sp,sp,-152
<6>346: frame_size 152
<6> SP 0x88171b18, RA 0x8010b4dc, PC 0x8010b4dc smp_call_function+0x1c0/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171b30, RA 0x8010b4dc, PC 0x8010b4dc smp_call_function+0x1c0/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171b48, RA 0x8010b4dc, PC 0x8010b4dc smp_call_function+0x1c0/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171b60, RA 0x8010b4dc, PC 0x8010b4dc smp_call_function+0x1c0/0x20c
<6>250: addiu sp,sp,-24
<6>346: frame_size 24
<6> SP 0x88171b78, RA 0x8010b4dc, PC 0x8010b4dc smp_call_function+0x1c0/0x20c
Photo of Alona

Alona, Employee

  • 1,700 Points 1k badge 2x thumb
This is just stack fuction call. Is this panic hapenning constantly - or you just have one time panic occurance? You can look at the timestamp and see how long ago panic occured. Again - you said MUs are not connecting and unless your controller is constantly rebooting - that panic is not what's causing you a problem. Probably something else in the config.