X670 logging

  • 0
  • 1
  • Question
  • Updated 2 years ago
  • Answered
It seems that problems within 2 X670V in Datacenter Distribution layer which working in a stack caused a network disruption (EXOS V 15.3.4.6). we using mLAG as redundancy mechanism to core switch and to the connected server - no STP. Unfortunately the system log tell me nothing about that problem.

So i do not know where is should looking for the root cause. If maybe there is a problem with switch stacking or hardware or only a network loop occurs (because of server misconfiguring or maybe a fault within mLAG)???

Are there any suggestions how to setup switch logging (EMS) mor granular. That if the next time a network disruption occours the logging allow a more details view ??

Thanks for helpful advices.
Photo of M.Nees

M.Nees, Embassador

  • 8,808 Points 5k badge 2x thumb

Posted 2 years ago

  • 0
  • 1
Photo of Kawawa

Kawawa, GTAC

  • 2,976 Points 2k badge 2x thumb
Hi Mathias,

In addition to the usual troubleshooting show commands, you can add events to the DefaulLogFilter or any custom logs you've defined that report on particular conditions, i.e. HAL.Stacking.Error would report if there is any stacking error, while events like HAL.VSM.AllocMLAGMemFail will let you know if a memory allocation for an MLAG operation failed, It all depends on what you want to see and the EMS Message Catalog is a good place to start and better understand what each component/subcomponent and matching conditions do.

Also, look at the following articles from out Knowledgebase:
How to troubleshoot MLAG issues
Useful logs filters can be added to track any event of stacking ports

If you have any specific information you're looking to understand, let us know and I am sure more specific things to look at can be recommended.  
Photo of M.Nees

M.Nees, Embassador

  • 8,808 Points 5k badge 2x thumb
Hi,

because in my special case i do not know where the issue is coming from i decide to set the error logging to nvram a little bit lower for hoping to see more than an event occurs.

configure log target nvram  filter DefaultFilter severity Warning (Default Value)
configure log target nvram  filter DefaultFilter severity Notice

is my assumption correct ?

How many log events can be stored in NVRAM ? Does i run into the problem that event logging (too much events) will waste / fill up switch nvram storage ? How can i check that ?


Regards
Photo of Kawawa

Kawawa, GTAC

  • 2,976 Points 2k badge 2x thumb
Hi, if I am not mistaken, the NVRAM log is only populated when the switch reboots and only with events with a severity level of warning and above.  The following command will give you an idea of what's currently configured for your actual log:
show log configuration filter "DefaultFilter"
By default, you can have 1,000 log entries, but you can increase that value to 20,000
show log configuration target memory-buffer
Log Target      : memory-buffer    Enabled     : yes
    Filter Name : DefaultFilter
    Match regex : Any
    Severity    : Debug-Data (through Critical)
    Format      : MM/DD/YYYY HH:MM:SS.hh <Severity:Component.SubComponent.Condition>
    Buffer size : 1000 messages
configure log target memory-buffer number-of-messages <value>
I am not sure if there is a counter to show how much of it you've used up, I am sure other colleagues can chip in on that one.  But when you display your log with the "show log" command, at the bottom there is a message "a total of %total% log messages were displayed"
Photo of EtherMAN

EtherMAN, Embassador

  • 6,306 Points 5k badge 2x thumb
why not just point your switches to a syslog server??? You will never miss a log entry due to nvram for the log filling up and you can use syslog to data mine for many things.  There are many good syslog server software out there including Netsight.  We us Kentik ourselves but there are many others.   
Photo of Chad Smith

Chad Smith, Senior Escalation Support Engineer

  • 5,620 Points 5k badge 2x thumb
Mathias,

Let me start by saying that it may be a good idea to open up a GTAC case so that we can help you investigate this problem in more detail.  

I would not recommend using the NVRAM buffer for this troubleshooting.  As EtherMAN has suggested, a syslog server would be perfect for this situation (How to configure a syslog server).

As for customizing the EMS log to make it more granular,  here are some tips:

The "show log events {all | <sub-componnent>}" command is helpful when trying to figure out what log messages are available (and their severity).  Listed below is an example of that command using vsm (the module that contains MLAG logging):

L3L_K1_U14(65.50).6 # show log events vsm
Component   SubComponent Condition               Severity      Parameters
----------- ------------ ----------------------- ------------- ----------
VSM                      AllocCkptMsgFail        Error          2 Total
VSM                      AllocHCMsgFail          Error          1 Total
VSM                      AllocMemFail            Error          2 Total
VSM                      AllocMSMCkptMsgFail     Error          2 Total
VSM                      AllocSockFail           Error          2 Total
VSM                      BlkMLAGTrace            Debug-Verbose  1 Total
VSM                      BndSockFail             Error          2 Total
VSM                      ConnCkptTrace           Debug-Verbose  1 Total
VSM                      FastCnvrgCtrlMLAGFail   Warning        1 Total
VSM                      FndISCIfFail            Warning        1 Total
VSM                      FndMLAGPortFail         Warning        1 Total
VSM                      InitSrvrDone            Info           0 Total
VSM                      InitSrvrFail            Error          0 Total
VSM                      LocMLAGPortDown         Info           3 Total
VSM                      LocMLAGPortUp           Info           3 Total
VSM                      MSMCkptTrace            Debug-Verbose  1 Total
VSM                      ParmInv                 Critical       2 Total
VSM                      ParmNull                Critical       1 Total
VSM                      RedirMLAGTrace          Debug-Verbose  1 Total
VSM                      RmtMLAGPeerDown         Info           2 Total
VSM                      RmtMLAGPeerUp           Info           2 Total
VSM                      RmtMLAGPortDown         Info           3 Total
VSM                      RmtMLAGPortUp           Info           3 Total
VSM                      RxCkptPkt               Debug-Verbose  5 Total
VSM                      RxCkptPktDump           Debug-Data     7 Total
VSM                      RxCkptPktInv            Error          4 Total
VSM                      RxCkptPktVerInv         Error          4 Total
VSM                      RxCkptSockFail          Warning        3 Total
VSM                      RxCkptTrace             Debug-Verbose  1 Total
VSM                      RxHCPkt                 Debug-Verbose  5 Total
VSM                      RxHCPktDump             Debug-Data     6 Total
VSM                      RxHCPktInv              Error          4 Total
VSM                      RxHCPktVerInv           Error          3 Total
VSM                      RxHCSockFail            Warning        1 Total
VSM                      RxHCTrace               Debug-Verbose  1 Total
VSM                      RxSrvrTrace             Debug-Verbose  1 Total
VSM                      SetSockFail             Error          3 Total
VSM                      TxCkptPkt               Debug-Verbose  5 Total
VSM                      TxCkptPktDump           Debug-Data     7 Total
VSM                      TxCkptSockFail          Warning        3 Total
VSM                      TxCkptTrace             Debug-Verbose  1 Total
VSM                      TxHCPkt                 Debug-Verbose  3 Total
VSM                      TxHCPktDump             Debug-Data     5 Total
VSM                      TxHCSockFail            Warning        3 Total
VSM                      TxHCTrace               Debug-Verbose  1 Total
VSM                      TxSrvrTrace             Debug-Verbose  1 Total
You can use the "show log counters {all | <subcomponent>} occured" command to display all log messages that have occurred.  Example using vsm (no MLAG is currently configured on this particular switch):

L3L_K1_U14(65.50).7 # show log counters vsm occurred
 Component   SubComponent Condition               Severity      Occurred I Ntfd
----------- ------------ ----------------------- ------------- -------- - ----
VSM                      InitSrvrDone            Info                 1 N    0
VSM                      RedirMLAGTrace          Debug-Verbose        3 N    0
The Occurred counter will let you know if the message has occurred (since last reboot or counters being cleared).  The "I" column will let you know if the message is included in the log filter and the Ntfd column will let you how many times the message was printed to the log.

If you find a message that seems useful or interesting in the above outputs, it can be added to the default filter (or whatever filter you desire) using the following command:

configure log filter <filter> add event <event> 

or if you want to add all events for a particularly module that are a certain severity and above, you can add the entire module (i.e vsm) and specify a severity:

configure log filter <filter> add event <module> severity <severity>

you can also match on a specific parameter within the log message by specifying the match condition:

configure log filter <filter> add event <event> match <match_parameter> <value>

Note: for some messages to be printed to the log you may need to enable log debug-mode (not recommended unless requested by GTAC).

For stacking and MLAG it would probably be a good idea to look at the vsm, NM, and HAL.Stacking messages.
Photo of M.Nees

M.Nees, Embassador

  • 8,808 Points 5k badge 2x thumb
Hi Chad, thanks for your troubleshooting tips! I will try this.