topic RE: X670 logging in ExtremeSwitching (Other)

X670 logging

M_Nees — Mon, 14 Mar 2016 15:07:00 GMT

It seems that problems within 2 X670V in Datacenter Distribution layer which working in a stack caused a network disruption (EXOS V 15.3.4.6). we using mLAG as redundancy mechanism to core switch and to the connected server - no STP. Unfortunately the system log tell me nothing about that problem.

So i do not know where is should looking for the root cause. If maybe there is a problem with switch stacking or hardware or only a network loop occurs (because of server misconfiguring or maybe a fault within mLAG)???

Are there any suggestions how to setup switch logging (EMS) mor granular. That if the next time a network disruption occours the logging allow a more details view ??

Thanks for helpful advices.

RE: X670 logging

Kawawa — Mon, 14 Mar 2016 16:36:00 GMT

Hi Mathias,

In addition to the usual troubleshooting show commands, you can add events to the DefaulLogFilter or any custom logs you've defined that report on particular conditions, i.e. HAL.Stacking.Error would report if there is any stacking error, while events like HAL.VSM.AllocMLAGMemFail will let you know if a memory allocation for an MLAG operation failed, It all depends on what you want to see and the EMS Message Catalog is a good place to start and better understand what each component/subcomponent and matching conditions do.

Also, look at the following articles from out Knowledgebase:
How to troubleshoot MLAG iss ues
Useful logs filters can be added to track any event of stacking ports

If you have any specific information you're looking to understand, let us know and I am sure more specific things to look at can be recommended.

RE: X670 logging

M_Nees — Tue, 15 Mar 2016 13:21:00 GMT

Hi,

because in my special case i do not know where the issue is coming from i decide to set the error logging to nvram a little bit lower for hoping to see more than an event occurs.

configure log target nvram filter DefaultFilter severity Warning (Default Value)configure log target nvram filter DefaultFilter severity Notice
is my assumption correct ?

How many log events can be stored in NVRAM ? Does i run into the problem that event logging (too much events) will waste / fill up switch nvram storage ? How can i check that ?

Regards

RE: X670 logging

Kawawa — Tue, 15 Mar 2016 15:16:00 GMT

Hi, if I am not mistaken, the NVRAM log is only populated when the switch reboots and only with events with a severity level of warning and above. The following command will give you an idea of what's currently configured for your actual log:
show log configuration filter "DefaultFilter"By default, you can have 1,000 log entries, but you can increase that value to 20,000
show log configuration target memory-buffer
Log Target : memory-buffer Enabled : yes
Filter Name : DefaultFilter
Match regex : Any
Severity : Debug-Data (through Critical)
Format : MM/DD/YYYY HH:MM:SS.hh
Buffer size : 1000 messagesconfigure log target memory-buffer number-of-messages I am not sure if there is a counter to show how much of it you've used up, I am sure other colleagues can chip in on that one. But when you display your log with the "show log" command, at the bottom there is a message "a total of %total% log messages were displayed"

RE: X670 logging

EtherMAN — Tue, 15 Mar 2016 21:50:00 GMT

why not just point your switches to a syslog server??? You will never miss a log entry due to nvram for the log filling up and you can use syslog to data mine for many things. There are many good syslog server software out there including Netsight. We us Kentik ourselves but there are many others.

RE: X670 logging

Chad_Smith1 — Wed, 16 Mar 2016 01:29:00 GMT

Mathias,

Let me start by saying that it may be a good idea to open up a GTAC case so that we can help you investigate this problem in more detail.

I would not recommend using the NVRAM buffer for this troubleshooting. As EtherMAN has suggested, a syslog server would be perfect for this situation (How to configure a syslog server).

As for customizing the EMS log to make it more granular, here are some tips:

The "show log events {all | }" command is helpful when trying to figure out what log messages are available (and their severity). Listed below is an example of that command using vsm (the module that contains MLAG logging):

L3L_K1_U14(65.50).6 # show log events vsm Component SubComponent Condition Severity Parameters
----------- ------------ ----------------------- ------------- ----------
VSM AllocCkptMsgFail Error 2 Total
VSM AllocHCMsgFail Error 1 Total
VSM AllocMemFail Error 2 Total
VSM AllocMSMCkptMsgFail Error 2 Total
VSM AllocSockFail Error 2 Total
VSM BlkMLAGTrace Debug-Verbose 1 Total
VSM BndSockFail Error 2 Total
VSM ConnCkptTrace Debug-Verbose 1 Total
VSM FastCnvrgCtrlMLAGFail Warning 1 Total
VSM FndISCIfFail Warning 1 Total
VSM FndMLAGPortFail Warning 1 Total
VSM InitSrvrDone Info 0 Total
VSM InitSrvrFail Error 0 Total
VSM LocMLAGPortDown Info 3 Total
VSM LocMLAGPortUp Info 3 Total
VSM MSMCkptTrace Debug-Verbose 1 Total
VSM ParmInv Critical 2 Total
VSM ParmNull Critical 1 Total
VSM RedirMLAGTrace Debug-Verbose 1 Total
VSM RmtMLAGPeerDown Info 2 Total
VSM RmtMLAGPeerUp Info 2 Total
VSM RmtMLAGPortDown Info 3 Total
VSM RmtMLAGPortUp Info 3 Total
VSM RxCkptPkt Debug-Verbose 5 Total
VSM RxCkptPktDump Debug-Data 7 Total
VSM RxCkptPktInv Error 4 Total
VSM RxCkptPktVerInv Error 4 Total
VSM RxCkptSockFail Warning 3 Total
VSM RxCkptTrace Debug-Verbose 1 Total
VSM RxHCPkt Debug-Verbose 5 Total
VSM RxHCPktDump Debug-Data 6 Total
VSM RxHCPktInv Error 4 Total
VSM RxHCPktVerInv Error 3 Total
VSM RxHCSockFail Warning 1 Total
VSM RxHCTrace Debug-Verbose 1 Total
VSM RxSrvrTrace Debug-Verbose 1 Total
VSM SetSockFail Error 3 Total
VSM TxCkptPkt Debug-Verbose 5 Total
VSM TxCkptPktDump Debug-Data 7 Total
VSM TxCkptSockFail Warning 3 Total
VSM TxCkptTrace Debug-Verbose 1 Total
VSM TxHCPkt Debug-Verbose 3 Total
VSM TxHCPktDump Debug-Data 5 Total
VSM TxHCSockFail Warning 3 Total
VSM TxHCTrace Debug-Verbose 1 Total
VSM TxSrvrTrace Debug-Verbose 1 TotalYou can use the "show log counters {all | } occured" command to display all log messages that have occurred. Example using vsm (no MLAG is currently configured on this particular switch):

L3L_K1_U14(65.50).7 # show log counters vsm occurred
Component SubComponent Condition Severity Occurred I Ntfd
----------- ------------ ----------------------- ------------- -------- - ----
VSM InitSrvrDone Info 1 N 0
VSM RedirMLAGTrace Debug-Verbose 3 N 0The Occurred counter will let you know if the message has occurred (since last reboot or counters being cleared). The "I" column will let you know if the message is included in the log filter and the Ntfd column will let you how many times the message was printed to the log.

If you find a message that seems useful or interesting in the above outputs, it can be added to the default filter (or whatever filter you desire) using the following command:

configure log filter add event

or if you want to add all events for a particularly module that are a certain severity and above, you can add the entire module (i.e vsm) and specify a severity:

configure log filter add event severity

you can also match on a specific parameter within the log message by specifying the match condition:

configure log filter add event match

Note: for some messages to be printed to the log you may need to enable log debug-mode (not recommended unless requested by GTAC).

For stacking and MLAG it would probably be a good idea to look at the vsm, NM, and HAL.Stacking messages.

RE: X670 logging

M_Nees — Wed, 16 Mar 2016 02:35:00 GMT

Hi Chad, thanks for your troubleshooting tips! I will try this.