Wireless (General)

Expand all | Collapse all

NAC appliance is red in console, but green in XMC

  • 1.  NAC appliance is red in console, but green in XMC

    Posted 05-25-2018 13:15

    Hello, team,

    after reboot NAC is red in Console, but green in XMC. As result, nothing works.

    I've read:

    https://extremeportal.force.com/ExtrArticleDetail?an=000077969
    https://extremeportal.force.com/ExtrArticleDetail?an=000078011
    https://extremeportal.force.com/ExtrArticleDetail?an=000063624

    Nothing helped me. Curious, that nacstatus says that everything is OK.

    root@nac.kafedra.local:/var/log$ nacstatus

    #-------------------------------------------------------------------------------
    # NAC Status
    #-------------------------------------------------------------------------------

    NAC Device Type: iav
    NAC Device Version: 7.1.1.9
    NAC OS Version: Ubuntu 12.04lts (64bit)
    Management IP: 192.168.1.201

    #-------------------------------------------------------------------------------
    # Configuration Details
    #-------------------------------------------------------------------------------

    | NAC Engine Information | Access Control Engine - NETSIGHTEVAL v.7.1.1.9 |
    | License Status | Valid License [netsighteval] (Evaluation period expires in 64 days) |
    | Hypervisor | Microsoft Hyper-V |
    | NAC Engine IP | 192.168.1.200 |
    | NetSight Server IP Address | 192.168.1.201 |
    | NAC Server Status | up, ready since Fri May 25 16:17:58 MSK 2018 |
    | NAC Up Time (HH:MM:SS.mmmm) | 00:26:19.143 |

    #-------------------------------------------------------------------------------
    # Resource Details
    #-------------------------------------------------------------------------------

    | CPU Usage | User=4.93%, System=1.75%, Niced=0.00%, Idle=93.32%, Total=6.68% |
    | Memory Usage | Used=83.96%, Free=16.04%, Total=7.78 GB |
    | Swap Space | Used=0.00%, Free=100.00%, Total=7.78 GB |
    | NAC Process | Heap=82.89%, Non-Heap=17.11%, Total=426.4 MB |
    | Available Space | Path=/, Free-Space=30Gb, Total-Space=35Gb |

    #-------------------------------------------------------------------------------
    # Status Details
    #-------------------------------------------------------------------------------

    | Statistic | Current | Maximum | Total | Max Reached |
    | _________________________________ | _______ | _______ | _____ | ____________________________ |
    | Authentication Requests | 0/min | 0/min | 0 | Not Available |
    | Authentication Successes | 0/min | 0/min | 0 | Not Available |
    | Authentication Failures | 0/min | 0/min | 0 | Not Available |
    | Radius Challenges | 0/min | 0/min | 0 | Not Available |
    | Invalid Authentication Requests | 0/min | 0/min | 0 | Not Available |
    | Duplicate Authentication Requests | 0/min | 0/min | 0 | Not Available |
    | Malformed Authentication Requests | 0/min | 0/min | 0 | Not Available |
    | Bad Authentication Requests | 0/min | 0/min | 0 | Not Available |
    | Dropped Radius Packets | 0/min | 0/min | 0 | Not Available |
    | Unknown Radius Types | 0/min | 0/min | 0 | Not Available |
    | Assessment Requests | 0/min | 0/min | 0 | Not Available |
    | Captive Portal Requests | 0/min | 15/min | 32 | Fri May 25 16:21:04 MSK 2018 |
    | Contact Lost Switches | 0 | 0 | | Not Available |
    | IP Resolution Failures | 0/min | 0/min | 0 | Not Available |
    | IP Resolution Timeouts | 0/min | 0/min | 0 | Not Available |
    | Connected Agents | 0 | 0 | | Not Available |
    | End-System Events | 0/min | 0/min | 0 | Not Available |
    | End-Systems One Day Count | 8 | 8 | | Fri May 25 16:18:04 MSK 2018 |
    | End-Systems Current Count | 8 | 8 | | Fri May 25 16:18:04 MSK 2018 |

    | NAC Manager Connection | down, ready, since Thu Jan 01 03:00:00 MSK 1970 |
    | General Message Counters | 0 sent, 12 dropped |
    | Event Message Status | normal mode, since Fri May 25 16:18:01 MSK 2018 |
    | Event Message Counters | 0 sent, 0 pending, 0 dropped |
    | Health Result Message Status | normal mode, since Fri May 25 16:18:01 MSK 2018 |
    | Health Result Message Counters | 0 sent, 0 pending, 0 dropped |
    | NAC-to-NAC Message Status | merging mode, since Fri May 25 16:18:01 MSK 2018 |
    | NAC-to-NAC Mergable Message Counters | 0 sent, 2 pending, 0 dropped |
    | NAC-to-NAC Normal Message Counters | 0 sent, 2 pending, 0 dropped |
    | Update Group Request Counters | 0 sent, 0 pending, 0 dropped |
    | Comm Error Reauthenticator Counters | 0 topic connection drops detected |
    | Agent Remote Scan Request Counters | 0 sent, 0 pending, 0 dropped |
    | Agent State Change Counters | 0 sent, 0 pending, 0 dropped |
    | Distributed Cache Publisher | sent: 0 bootstrap requests |
    | Distributed Cache Subscriber | received: 0 activity messages, 0 activity events, 0 bootstrap messages, 0 bootstrap elements |
    | Distributed Cache Contents | 'EndSystem' (0) |
    | NAC Web Service Client | up, ready, since Fri May 25 16:29:09 MSK 2018 |
    | NAC AAA Thread Counter | Thread[NAC AAA Server Request Processor (127.0.0.1 port:1300),7,NacAAARequestHandler Group](ThreadGroup: 9), Max: 8 @ Fri May 25 16:24:00 MSK 2018 |
    | NAC ACCT Thread Counter | Thread[NAC ACCT Server Request Processor (127.0.0.1 port:1302),7,NacACCTRequestHandler Group](ThreadGroup: 5), Max: 4 @ Fri May 25 16:24:00 MSK 2018 |
    | Last Request Processed | Thu Jan 01 03:00:00 MSK 1970 |
    | Throttled Radius Requests | 0 |
    | NetBIOS Requests | 0 |

    #-------------------------------------------------------------------------------
    # NAC Thread Pool Details
    #-------------------------------------------------------------------------------

    | Thread Name | Active Count | Pool Size | Queue Size | Max Queue Size | Queue Limit Reached | Throttled Tasks | Tasks Completed |
    | ________________________________________________________ | ____________ | _________ | __________ | ______________ | ___________________ | _______________ | _______________ |
    | Assessment Controller Thread Pool | 0 | 10 | 0 | 12000 | | 0 | 0 |
    | EnforceHandler - Notify Listeners Thread Pool | 0 | 1 | 0 | 12000 | | 0 | 18 |
    | EnforceHandler - Off Thread Notify Listeners Thread Pool | 0 | 1 | 0 | 12000 | | 0 | 3 |
    | Initialize Switch Thread Thread Pool | 0 | 20 | 0 | 12000 | | 0 | 1 |
    | NAC 2 NAC Message Handler Thread Pool | 0 | 1 | 0 | 10000 | | 0 | 74 |
    | NAC Manager Config Message Handler Thread Pool | 0 | 1 | 0 | 12000 | | 0 | 0 |
    | NAC Manager Status Message Handler Thread Pool | 0 | 1 | 0 | 12000 | | 0 | 0 |
    | NAC Status Request Executor Thread Pool | 0 | 1 | 0 | 12000 | | 0 | 0 |
    | NacCaptivePortalMainAction - Task Thread Pool | 0 | 10 | 0 | 12000 | | 0 | 0 |
    | NetBIOS Request Manager Thread Pool | 0 | 5 | 0 | 500 | | 0 | 0 |
    | RADIUS Session Deactivate Queue Thread Pool | 0 | 1 | 0 | 12000 | | 0 | 0 |
    | SNMP Manager Refresh Child Thread Pool | 0 | 1 | 0 | 12000 | | 0 | 0 |
    | SNMP Manager Refresh Parent Thread Pool | 0 | 1 | 0 | 12000 | | 0 | 0 |
    | Switch Configuration Thread Pool | 0 | 1 | 0 | 10000 | | 0 | 2 |
    | Switch Configuration Scheduled Thread Pool | 0 | 1 | 0 | 10000 | | 0 | 1 |
    | Switch Configuration Task Thread Pool | 0 | 10 | 0 | 10000 | | 0 | 1 |
    | TopicSubPub MessageMaker Thread Pool | 0 | 2 | 0 | 12000 | | 0 | 0 |

    #-------------------------------------------------------------------------------
    # NetSight Server Name Resolution
    #-------------------------------------------------------------------------------

    Resolving NetSight Server Name: NetSight
    Server: 192.168.1.2
    Address: 192.168.1.2#53

    Name: NetSight.kafedra.local
    Address: 192.168.1.201

    #-------------------------------------------------------------------------------
    # NAC Server Name Resolution
    #-------------------------------------------------------------------------------

    Resolving NAC Server Name: nac.kafedra.local
    Server: 192.168.1.2
    Address: 192.168.1.2#53

    Name: nac.kafedra.local
    Address: 192.168.1.200

    #-------------------------------------------------------------------------------
    # Communications Diagnostics
    #-------------------------------------------------------------------------------

    NAC to NetSight WebServices: SUCCESS.
    NetSight to NAC Appliance WebServices: SUCCESS.
    JMS Topic Connection: DOWN.
    NetSight Server IP: 192.168.1.201
    DNS Server IP: 192.168.1.2
    NAC Domain Name: kafedra.local
    Reverse DNS Lookup Timeout: 10
    Reverse DNS Lookup of NAC Address: netsight (< 1 sec)
    NAC Registration and Remediation IP: 192.168.1.200
    NAC Hostname DNS Resolution: 192.168.1.200



  • 2.  RE: NAC appliance is red in console, but green in XMC

    Posted 05-28-2018 12:05
    Hello,

    First i'd make sure that you're not seeing an active alarm. Can you make sure and clear the alarms on the appliance?

    Next, in NetSight Console right click on the NAC appliance and chose "MIB tools".

    Does the bottom bar on the MIB tools window show an error like "Authentication failed"?

    Check and make sure that the NAC has the correct profile, with the correct authentication/privacy parameters.

    It needs to be set to auth/priv, and the credentials can be checked/reconfigured by running the "nacconfig" command on the NAC appliance itself.

    Thanks
    -Ryan



  • 3.  RE: NAC appliance is red in console, but green in XMC

    Posted 08-14-2018 04:20
    Hi Ilya,

    have you solved the issue?
    I'm currently having same behavior at one of my customers.


  • 4.  RE: NAC appliance is red in console, but green in XMC

    Posted 05-28-2018 12:05
    Hello, Ryan,

    I've done a new installation. Everything is 8.1. Now under ESXi 6.0. All VMs are in the same subnet, no firewall between them.

    Both NACs are green in XMC, but red in NAC console. They all ping each other by IPs and hostnames.



    I can see NAC's MIB Profiles on Netsight Console, no errors.



    I use default snmp_v3_profile, everything is AuthPriv, and I left exactly these settings during appliance installation:



    Nacstatus on both appliances says:

    root@nac1.spbstu.ru:~$ nacstatus

    #-------------------------------------------------------------------------------
    # NAC Status
    #-------------------------------------------------------------------------------

    NAC Device Type: iav
    NAC Device Version: 8.1.2.60
    NAC OS Version: Ubuntu 14.04lts (64bit)
    Management IP: 192.168.245.184

    #-------------------------------------------------------------------------------
    # Configuration Details
    #-------------------------------------------------------------------------------

    | EAC Engine Information | Access Control Engine - IA-V v.8.1.2.60 |
    | License Status | No License - this appliance will not operate without a valid license |
    | Hypervisor | VMWare ESX (0xEA580) |
    | Extreme Access Control(EAC) Engine IP | 192.168.245.185 |
    | Extreme Management Server IP Address | 192.168.245.184 |
    | EAC Server Status | up, ready since Fri Jun 08 14:42:04 MSK 2018 |
    | EAC Up Time (HH:MM:SS.mmmm) | 00:22:42.158 |

    #-------------------------------------------------------------------------------
    # Resource Details
    #-------------------------------------------------------------------------------

    | CPU Usage | User=18.59%, System=1.10%, Niced=0.00%, Idle=80.31%, Total=19.69% |
    | Memory Usage | Used=10.44%, Free=89.56%, Total=11.73 GB |
    | Swap Space | Used=0.00%, Free=100.00%, Total=11.73 GB |
    | EAC Process | Heap=75.39%, Non-Heap=24.61%, Total=268.72 MB |
    | Available Space | Path=/, Free-Space=22Gb, Total-Space=27Gb |

    #-------------------------------------------------------------------------------
    # Status Details
    #-------------------------------------------------------------------------------

    | Statistic | Current | Maximum | Total | Max Reached |
    | _________________________________ | _______ | _______ | _____ | _____________ |
    | Authentication Requests | 0/min | 0/min | 0 | Not Available |
    | Authentication Successes | 0/min | 0/min | 0 | Not Available |
    | Authentication Failures | 0/min | 0/min | 0 | Not Available |
    | Radius Challenges | 0/min | 0/min | 0 | Not Available |
    | Invalid Authentication Requests | 0/min | 0/min | 0 | Not Available |
    | Duplicate Authentication Requests | 0/min | 0/min | 0 | Not Available |
    | Malformed Authentication Requests | 0/min | 0/min | 0 | Not Available |
    | Bad Authentication Requests | 0/min | 0/min | 0 | Not Available |
    | Dropped Radius Packets | 0/min | 0/min | 0 | Not Available |
    | Unknown Radius Types | 0/min | 0/min | 0 | Not Available |
    | Assessment Requests | 0/min | 0/min | 0 | Not Available |
    | Captive Portal Requests | 0/min | 0/min | 0 | Not Available |
    | Contact Lost Switches | 0 | 0 | | Not Available |
    | IP Resolution Failures | 0/min | 0/min | 0 | Not Available |
    | IP Resolution Timeouts | 0/min | 0/min | 0 | Not Available |
    | Connected Agents | 0 | 0 | | Not Available |
    | End-System Events | 0/min | 0/min | 0 | Not Available |
    | End-Systems One Day Count | 0 | 0 | | Not Available |
    | End-Systems Current Count | 0 | 0 | | Not Available |

    | EAC Manager Connection | down, not ready, since Thu Jan 01 03:00:00 MSK 1970 |
    | General Message Counters | 0 sent, 9 dropped |
    | Event Message Status | normal mode, since Fri Jun 08 14:42:08 MSK 2018 |
    | Event Message Counters | 0 sent, 0 pending, 0 dropped |
    | Health Result Message Status | normal mode, since Fri Jun 08 14:42:08 MSK 2018 |
    | Health Result Message Counters | 0 sent, 0 pending, 0 dropped |
    | EAC-to-EAC Message Status | merging mode, since Fri Jun 08 14:42:08 MSK 2018 |
    | EAC-to-EAC Mergable Message Counters | 0 sent, 1 pending, 0 dropped |
    | EAC-to-EAC Normal Message Counters | 0 sent, 1 pending, 0 dropped |
    | Update Group Request Counters | 0 sent, 0 pending, 0 dropped |
    | Comm Error Reauthenticator Counters | 0 topic connection drops detected |
    | Agent Remote Scan Request Counters | 0 sent, 0 pending, 0 dropped |
    | Agent State Change Counters | 0 sent, 0 pending, 0 dropped |
    | EAC Web Service Client | down, not ready, since Fri Jun 08 14:42:13 MSK 2018 |
    | EAC AAA Thread Counter | Thread[EAC AAA Server Request Processor (127.0.0.1 port:1300),7,EacAAARequestHandler Group](ThreadGroup: 2), Max: 1 @ Fri Jun 08 14:42:22 MSK 2018 |
    | EAC ACCT Thread Counter | Thread[EAC ACCT Server Request Processor (127.0.0.1 port:1302),7,EacACCTRequestHandler Group](ThreadGroup: 1), Max: 0 @ Thu Jan 01 03:00:00 MSK 1970 |
    | Last Request Processed | Thu Jan 01 03:00:00 MSK 1970 |
    | Throttled Radius Requests | 0 |
    | NetBIOS Requests | 0 |

    #-------------------------------------------------------------------------------
    # EAC Thread Pool Details
    #-------------------------------------------------------------------------------

    | Thread Name | Active Count | Pool Size | Queue Size | Max Queue Size | Queue Limit Reached | Throttled Tasks | Tasks Completed |
    | ________________________________________________________ | ____________ | _________ | __________ | ______________ | ___________________ | _______________ | _______________ |
    | Assessment Controller Thread Pool | 0 | 10 | 0 | 6000 | | 0 | 0 |
    | EAC 2 EAC Message Handler Thread Pool | 0 | 1 | 0 | 10000 | | 0 | 31 |
    | EAC Manager Config Message Handler Thread Pool | 0 | 1 | 0 | 6000 | | 0 | 0 |
    | EAC Manager Status Message Handler Thread Pool | 0 | 1 | 0 | 6000 | | 0 | 0 |
    | EAC Status Request Executor Thread Pool | 0 | 1 | 0 | 6000 | | 0 | 0 |
    | EnforceHandler - Off Thread Notify Listeners Thread Pool | 0 | 1 | 0 | 6000 | | 0 | 1 |
    | Initialize Switch Thread Thread Pool | 0 | 20 | 0 | 6000 | | 0 | 0 |
    | NetBIOS Request Manager Thread Pool | 0 | 5 | 0 | 500 | | 0 | 0 |
    | SNMP Manager Refresh Child Thread Pool | 0 | 1 | 0 | 6000 | | 0 | 0 |
    | SNMP Manager Refresh Parent Thread Pool | 0 | 1 | 0 | 6000 | | 0 | 0 |
    | Switch Configuration Scheduled Thread Pool | 0 | 1 | 0 | 10000 | | 0 | 1 |
    | TopicSubPub MessageMaker Thread Pool | 0 | 2 | 0 | 6000 | | 0 | 0 |

    #-------------------------------------------------------------------------------
    # Startup End-System Auth Count Information
    #-------------------------------------------------------------------------------

    Current End-System count from last day at startup is: 0

    Current active (not disconnected) End-System count at startup is: 0

    Totals State - Accept: 0, Reject: 0, Scan: 0, Quarantine: 0, Error: 0, Disconnected: 0

    Totals ConnectedState - Active: 0, Active with Highest Precedence: 0, Disconnected: 0, Unknown: 0

    #-------------------------------------------------------------------------------
    # NetSight Server Name Resolution
    #-------------------------------------------------------------------------------

    #-------------------------------------------------------------------------------
    # NAC Server Name Resolution
    #-------------------------------------------------------------------------------

    Resolving NAC Server Name: nac1.spbstu.ru
    Server: 194.190.225.225
    Address: 194.190.225.225#53

    Name: nac1.spbstu.ru
    Address: 192.168.245.185

    #-------------------------------------------------------------------------------
    # Communications Diagnostics
    #-------------------------------------------------------------------------------

    NAC to NetSight WebServices: FAILURE.
    NetSight to NAC WebServices: UNABLE TO TEST.
    JMS Topic Connection: DOWN.
    NetSight Server IP: 192.168.245.184
    DNS Server IP: 194.190.225.225
    NAC Domain Name: spbstu.ru
    Reverse DNS Lookup Timeout: 10
    Reverse DNS Lookup of NAC Address: xmc.spbstu.ru (< 1 sec)
    NAC Registration and Remediation IP: 192.168.245.185
    NAC Hostname DNS Resolution: 192.168.245.185

    #-------------------------------------------------------------------------------
    # Appliance License and Capacity Diagnostics
    #-------------------------------------------------------------------------------

    NAC appliance is virtual.
    Virtual NAC appliance is not licensed.
    License Status: No License
    License Data: null
    Current End-System Capacity: 2000
    Assessment Capable: False

    #-------------------------------------------------------------------------------
    # Distributed Cache Diagnostics
    #-------------------------------------------------------------------------------

    NAC appliance distributed cache is disabled.
    Distributed Caches found: 0
    Distributed Caches Counters:
    + Bootstrap Requests Sent: 0
    + Bootstrap Messages Received: 0
    + Bootstrap Elements Received: 0
    + Activity Messages Received: 0
    + Activity Events Received: 0

    #-------------------------------------------------------------------------------
    # Process Status
    #-------------------------------------------------------------------------------

    EAC Watchdog Process Check Success
    Database process is running.
    RADIUS Process is running.
    EAC Process is running.

    #-------------------------------------------------------------------------------
    # Most Recent Errors from /var/log/syslog
    #-------------------------------------------------------------------------------

    Jun 8 14:21:44 nac1 kernel: [ 4.481702] EXT4-fs (dm-0): re-mounted. Opts: errors=remount-ro

    #-------------------------------------------------------------------------------
    # Most Recent Actions from /var/log/watchdog.log
    #-------------------------------------------------------------------------------

    2018-06-08 14:21:48,374 INFO [SyslogWriter] Watchdog Service is starting
    2018-06-08 14:42:03,596 INFO [SyslogWriter] Watchdog Service is starting

    #-------------------------------------------------------------------------------
    # Most Recent Errors from /var/log/tag.log
    #-------------------------------------------------------------------------------

    #-------------------------------------------------------------------------------
    # Most Recent Errors from /var/log/radius/radius.log
    #-------------------------------------------------------------------------------

    Thu Jun 7 15:52:20 2018 : Error: [etsnac connection_mgr] Failed to connect to server 127.0.0.1 on port: 1300 with error: Connection refused(111)
    Thu Jun 7 16:07:22 2018 : Error: [etsnac connection_mgr] Failed to connect to server 127.0.0.1 on port: 1300 with error: Connection refused(111)
    Fri Jun 8 14:21:47 2018 : Error: [etsnac connection_mgr] Failed to connect to server 127.0.0.1 on port: 1300 with error: Connection refused(111)
    Fri Jun 8 14:42:02 2018 : Error: [etsnac connection_mgr] Failed to connect to server 127.0.0.1 on port: 1300 with error: Connection refused(111)

    #-------------------------------------------------------------------------------
    # ProxyRedirect status
    #-------------------------------------------------------------------------------

    ProxyRedirector threads running: 0

    #-------------------------------------------------------------------------------
    # Squid Status
    #-------------------------------------------------------------------------------

    ERROR: Cannot connect to 127.0.0.1:3128

    #-------------------------------------------------------------------------------
    # NetSight server status
    #-------------------------------------------------------------------------------

    Checking Status of Access Control Engine, RADIUS, Proxy & Agentless Assessment Server:
    Access Control Engine Proxy is NOT running...
    Access Control Engine Server is running with PID: 4659
    Access Control Engine RADIUS Server is running with PID: 4633
    Agentless Assessment Server is running with PID: 4561

    Run '/sbin/nacctl restart'.

    #-------------------------------------------------------------------------------
    # Hostname Information
    #-------------------------------------------------------------------------------

    Hostname: nac1.spbstu.ru
    ################################################################################
    ## hosts - hosts - local host configuration file
    ##
    ## WARNING: This file is automatically generated on every enforce.
    ## This file is made from the following templates. Any modifications
    ## should be made to the template files, not this file.
    ##
    ## templates/hosts.tpl
    ##
    ################################################################################

    #
    # hosts


  • 5.  RE: NAC appliance is red in console, but green in XMC

    Posted 05-28-2018 12:05
    Hello,

    I would advise creation of a case for further investigation.

    NAC to NetSight WebServices: FAILURE.
    NetSight to NAC WebServices: UNABLE TO TEST.
    JMS Topic Connection: DOWN.

    Thanks
    -Ryan



  • 6.  RE: NAC appliance is red in console, but green in XMC

    Posted 08-14-2018 04:20
    Eventually, I've made a clean install of NAC. That's solved the issue. Time sync between EWC and NAC is a very important thing also - you should setup same NTP settings everywhere.