How to Access the DFE Fault Log

  • 0
  • 1
  • Article
  • Updated 5 years ago
  • (Edited)
Article ID: 5101 

Products
Matrix N-Series
DFE 

Related to
Faultlog
Fault log
Message log
atgtools 

Goals
Access the DFE Fault / Message log 

Solution
The Distributed Forwarding Engine (DFE) is a tightly integrated, chassis-based product. Under normal circumstances, the entire chassis is functionally considered to be a single switching unit. 

An aggregated Fault Log for the entire chassis is available in a single operation via MIB etsysDiagnosticMessageMIB=1.3.6.1.4.1.5624.1.2.13. There are two ways of rendering that information in a preformatted form:
  • Using "ATG Tools" as explained in this solution, the data is usefully organized for diagnostic purposes.
  • Using a NetSight Atlas flexview, the resulting data is not optimally sorted and is missing some key diagnostic information (5497).

With firmware 5.31.17 and higher, a good alternative to SNMP-based extraction is the use of the text output of the 'show support' command. 

In situations involving chassis connectivity problems, and/or blades that fail to boot to a fully operational state, none of these system-based procedures will produce a complete record; so the Message Log(s) for the non-communicating blade(s) must be individually extracted by means of the serial console. 

Both the system-based and blade-based methods of ATG Tools log extraction are explained below. 

Any Fault/Message Logs submitted to Enterasys Networks for analysis must be complete (every message and every slot), extracted in file format as described below, and forwarded to Enterasys Networks in that format. A cut-and-paste methodology is not acceptable. Engineering will not respond to log fragments or logs that are not submitted in the proper form. 


Important Note: Of the great variety of information that is viewable in a DFE Fault/Message Log, much of it is routine diagnostics, and does not indicate any real problem. In the presence of hardware or software issues, error reporting will also appear, as with the earlier SmartSwitch products. 

Neither message type is clearly identified as such, or is necessarily documented in our Knowledgebase. Unless a Knowledgebase solution states otherwise, it should be assumed that any given message is of the routine diagnostic variety, and is thus of no specific concern. 

In the presence of a DFE issue, it is appropriate to review the Fault/Message Logs for some type of accompanying error diagnostic; but attempting to determine the potential presence or absence of issues by a review of the Fault/Message Logs is generally not a practical plan. 


Extracting a Fault Log from a DFE Chassis 

How to extract a Fault Log from the DFE chassis, using ATG Tools:
  1. Download ATG Tools, available on our support website.
     
  2. Install it by placing the downloaded file into your root directory, and unzipping it while preserving the zipped directory structure (this is a WinZip option); resulting in a new c:\atgtools directory.
     
  3. Create a shortcut to c:\atgtools\doc\manual_atgtools.html, which can be executed at any time if the extra assistance of a web-based help screen becomes necessary.
     
  4. Run the getFaultLog option from a DOS prompt: Change to root directory ("cd \"), run the utility (e.g. "atgtools\bin\getfaultlog -s 10.1.1.3"), then exit when it has completed ("exit").
      Note: The getfaultlog tool supports only SNMPv1. If instead of the above result you receive error message "<ip_address>: is not alive" or similar, this is possibly because default SNMPv1 community name 'public' has been disabled. If a valid SNMPv1 community name exists, append the community name parameter ' -n <community_name>' to the getfaultlog command string; otherwise first use 5245 as a guide to temporarily permit SNMPv1 access. Long-term, the best way to generally disable (non-encrypted, unsecure) SNMPv1 is to merely issue the command 'clear snmp community <community_name>' against the SNMPv1 community name, while leaving the supporting SNMPv1 structure in place. Then, any time it is temporarily needed again for any purpose (including this one), issue the command 'set snmp community <community_name>'. In this manner SNMPv1 access may be quickly disabled or enabled.
       
    • The resulting getFaultLog.out file will reside in the same directory from which the command was executed. In this case, the path is c:\getFaultLog.out. 

    • Open it with WordPad, or equivalent text editor program. This can be a very lengthy file, for three primary reasons:
      1. The DFE switches log many more specific and routine events than do the earlier products.
      2. The Fault Log ideally should never be cleared, so that the audit trail will not be lost.
      3. The data is extracted from all of the DFE switch units (1-7) present in the queried N3/N7/E7 chassis.
    Shown below is the beginning portion of a Fault Log as extracted by this method. Fault Log output data from each slot can be differentiated by means of the (left-most) 'Index' column. The Index number uses the first digit for slot# and the remaining digits as a message sequence# (in ascending sequence by date/time). Sorting by index number means the items are in order by slot number, then by date/time of occurrence:

    =================================================================================
    =================================================================================

    IP Address: 10.1.1.3
    Description: Enterasys Networks, Inc. Matrix E7 7x4xxx Rev 01.08.02 06/18/2003--23:08 ofc
    Uptime: 06d_07:32:13.65

    ---------------------------------------------------------------------------------------
    Index ModelName HardwareRev FirmwareRev SoftwareRev SerialNum Alias AssetID
    ---------------------------------------------------------------------------------------
    72 7H4383-49 0 01.00.07 01.08.02 CI-A2
    73 7G4202-30 0 01.00.04 01.08.02 GR-A8
    74 7H4382-49 4 01.00.03 01.08.02 TRI-2R130
    75 7H4382-49 2 01.00.03 01.08.02 TRI-A26
    76 7G4270-12 1 01.00.03 01.08.02 CH-RA45
    77 7G4202-30 0 01.00.05 01.08.02 GR-A2

    Fault Log:

    ------------------------------------------------------------------------------------------------- -------
    Index Time Type FWRevision Summary
    ------------------------------------------------------------------------------------------------- -------
    2001 2003-5-16,10:2:9.0 Diagnostic Init 01.60.00 Initialization Complete

    The software system has successfully completed initialization
    and is entering its diagnostic operational mode.

    2002 2003-5-16,10:6:57.0 Diagnostic Info 01.60.00 Front Panel Ports

    Device Name: Front Panel Ports
    Test Name: Link Check Test
    Test Result: Failed
    Error Description: Link has not been detected on one or more ports

    Extracting a Message Log from a DFE blade 

    How to extract a Message Log from a DFE blade (14837), using either HyperTerminal or TeraTerm. This serial process must be repeated for each slot that is isolated from chassis communication or fails to boot to a fully operational state:
    1. Using one of these two applications, connect to the device serially (a Telnet session will not work). Use a standard "PC Adapter" (from kit 9372065), straight-through RJ45 cable, and settings of 9600/8/1/none, flow control 'Xon/Xoff' or 'none'.
       
    2. Reboot the blade.
       
    3. Boot messages will appear, as in the following example:

      Boot ROM Initialization, Version 01.00.01

      Copyright (c) 2003 Enterasys Networks, Inc.


      SDRAM size: 128 MB
      Testing SDRAM.... PASSED.

      Loading Boot Image: 01.00.03... DONE.

      Uncompressing Boot Image... DONE.

      Press any key to enter System Image Loader menu

    4. At this point, you have about two seconds to press a key (such as the space bar) to pause the boot process. If you fail to do so, it will be necessary to go back to step 2.
       
    5. A "[System Image Loader]:" prompt will appear. Type "log" (without the quotes) and hit <enter>.
       
    6. A summary Message Log will appear, as in the following example:

      1 06/23/2003 16:04:17 Shutdown Completed - Reset
      2 06/23/2003 16:03:17 Shutdown Initiated - Reset
      3 06/23/2003 16:03:14 Informational Received Push Button Reset.
      4 06/23/2003 16:02:36 System Init Initialization Complete
      5 06/23/2003 16:01:55 Shutdown Completed - Reset
      6 06/23/2003 16:00:54 Shutdown Initiated - Reset
      7 06/23/2003 16:00:51 Informational Received Push Button Reset.
      8 06/18/2003 12:39:39 System Init Initialization Complete
      9 06/18/2003 12:39:09 Shutdown Completed - Reset
      10 06/18/2003 12:39:04 Shutdown Initiated - Reset
      11 06/18/2003 12:39:02 Unknown Msg Type 3c 33 3e 46 69 6c 65 4d 67 72
      12 06/18/2003 12:35:06 System Init Initialization Complete
      13 06/18/2003 12:34:24 Shutdown Completed - Reset
      14 06/18/2003 12:34:19 Shutdown Initiated - Reset

      MessageLog >

    7. Set up and capture the detail Message Log to a text file:
       
      HyperTerminal:
      1. Using the left mouse button, click on the 'Transfer' dropdown menu option, on the menu bar at the top of the screen.
      2. Select the 'Capture Text' sub-option.
      3. In the launched "Capture Text" window, enter the destination directory and file name (ex: myfile.txt) for the text capture.
      4. Click the 'Start' button, starting the capture.
      5. In the Telnet session's "MessageLog >" prompt, type "a" (without the quotes) and hit <enter>.
      6. The entire Log will scroll by on the screen. Wait until it is complete and has stopped.
      7. Click on the 'Transfer' dropdown menu option.
      8. Select the 'Stop' sub-option. This ends the capture, returning to the active Hyperterm session.
      9. In the file location originally specified, the output text file is now present.
       
      TeraTerm:
      1. Using the left mouse button, click on the 'File' dropdown menu option, on the menu bar at the top of the screen.
      2. Select the 'Log' sub-option.
      3. In the launched "Tera Term: Log" window, enter the destination directory and file name (ex: myfile.txt) for the text capture.
      4. Click the 'Open' button, starting the capture.
      5. In the Telnet session's "MessageLog >" prompt, type "a" (without the quotes) and hit <enter>.
      6. The entire Log will scroll by on the screen. Wait until it is complete and has stopped.
      7. Click on the 'File' dropdown menu option.
      8. Select the 'Disconnect' sub-option. At the "Disconnect?" prompt, click on the 'OK' button. This not only ends the capture, but shuts down the TeraTerm session and application.
      9. In the file location originally specified, the output text file is now present.
       
    8. Open it with WordPad, or equivalent text editor program. This can be a very lengthy file, for two primary reasons:
      1. The DFE switches log many more specific and routine events than do the earlier products.
      2. The Message Log ideally should never be cleared, so that the audit trail will not be lost.
    Shown below is the beginning portion of a Message Log as extracted by this method:

    ================================================================================
    System Information
    ================================================================================
    Firmware Revision: None
    BootPROM Revision: 01.00.03
    Slot Number: 1
    Base MAC Address: 0001f427d335 IP Address: 0.0.0.0
    Current Time: 06/23/2003 16:05:01
    ==============================================================================
    Message 1/98 Shutdown 02.00.02 06/23/2003 16:04:17

    Completed - Reset

    More info details will come to COMPLETED_RESET!


    ==============================================================================
    Message 2/98 Shutdown 02.00.02 06/23/2003 16:03:17

    Initiated - Reset
    More info details will come to INITIATED_RESET!


    ==============================================================================
    Photo of FAQ User

    FAQ User, Official Rep

    • 13,610 Points 10k badge 2x thumb

    Posted 5 years ago

    • 0
    • 1

    There are no replies.

    This conversation is no longer open for comments or replies.