topic RE: NAC appliance is red in console, but green in XMC in ExtremeWireless (General)

NAC appliance is red in console, but green in XMC

Ilya_Semenov — Fri, 25 May 2018 20:15:00 GMT

Hello, team,

after reboot NAC is red in Console, but green in XMC. As result, nothing works.

I've read:

https://extremeportal.force.com/ExtrArticleDetail?an=000077969
https://extremeportal.force.com/ExtrArticleDetail?an=000078011
https://extremeportal.force.com/ExtrArticleDetail?an=000063624

Nothing helped me. Curious, that nacstatus says that everything is OK.

root@nac.kafedra.local:/var/log$ nacstatus

#-------------------------------------------------------------------------------
# NAC Status
#-------------------------------------------------------------------------------

NAC Device Type: iav
NAC Device Version: 7.1.1.9
NAC OS Version: Ubuntu 12.04lts (64bit)
Management IP: 192.168.1.201

#-------------------------------------------------------------------------------
# Configuration Details
#-------------------------------------------------------------------------------

| NAC Engine Information | Access Control Engine - NETSIGHTEVAL v.7.1.1.9 |
| License Status | Valid License [netsighteval] (Evaluation period expires in 64 days) |
| Hypervisor | Microsoft Hyper-V |
| NAC Engine IP | 192.168.1.200 |
| NetSight Server IP Address | 192.168.1.201 |
| NAC Server Status | up, ready since Fri May 25 16:17:58 MSK 2018 |
| NAC Up Time (HH:MM:SS.mmmm) | 00:26:19.143 |

#-------------------------------------------------------------------------------
# Resource Details
#-------------------------------------------------------------------------------

| CPU Usage | User=4.93%, System=1.75%, Niced=0.00%, Idle=93.32%, Total=6.68% |
| Memory Usage | Used=83.96%, Free=16.04%, Total=7.78 GB |
| Swap Space | Used=0.00%, Free=100.00%, Total=7.78 GB |
| NAC Process | Heap=82.89%, Non-Heap=17.11%, Total=426.4 MB |
| Available Space | Path=/, Free-Space=30Gb, Total-Space=35Gb |

#-------------------------------------------------------------------------------
# Status Details
#-------------------------------------------------------------------------------

| Statistic | Current | Maximum | Total | Max Reached |
| _________________________________ | _______ | _______ | _____ | ____________________________ |
| Authentication Requests | 0/min | 0/min | 0 | Not Available |
| Authentication Successes | 0/min | 0/min | 0 | Not Available |
| Authentication Failures | 0/min | 0/min | 0 | Not Available |
| Radius Challenges | 0/min | 0/min | 0 | Not Available |
| Invalid Authentication Requests | 0/min | 0/min | 0 | Not Available |
| Duplicate Authentication Requests | 0/min | 0/min | 0 | Not Available |
| Malformed Authentication Requests | 0/min | 0/min | 0 | Not Available |
| Bad Authentication Requests | 0/min | 0/min | 0 | Not Available |
| Dropped Radius Packets | 0/min | 0/min | 0 | Not Available |
| Unknown Radius Types | 0/min | 0/min | 0 | Not Available |
| Assessment Requests | 0/min | 0/min | 0 | Not Available |
| Captive Portal Requests | 0/min | 15/min | 32 | Fri May 25 16:21:04 MSK 2018 |
| Contact Lost Switches | 0 | 0 | | Not Available |
| IP Resolution Failures | 0/min | 0/min | 0 | Not Available |
| IP Resolution Timeouts | 0/min | 0/min | 0 | Not Available |
| Connected Agents | 0 | 0 | | Not Available |
| End-System Events | 0/min | 0/min | 0 | Not Available |
| End-Systems One Day Count | 8 | 8 | | Fri May 25 16:18:04 MSK 2018 |
| End-Systems Current Count | 8 | 8 | | Fri May 25 16:18:04 MSK 2018 |

| NAC Manager Connection | down, ready, since Thu Jan 01 03:00:00 MSK 1970 |
| General Message Counters | 0 sent, 12 dropped |
| Event Message Status | normal mode, since Fri May 25 16:18:01 MSK 2018 |
| Event Message Counters | 0 sent, 0 pending, 0 dropped |
| Health Result Message Status | normal mode, since Fri May 25 16:18:01 MSK 2018 |
| Health Result Message Counters | 0 sent, 0 pending, 0 dropped |
| NAC-to-NAC Message Status | merging mode, since Fri May 25 16:18:01 MSK 2018 |
| NAC-to-NAC Mergable Message Counters | 0 sent, 2 pending, 0 dropped |
| NAC-to-NAC Normal Message Counters | 0 sent, 2 pending, 0 dropped |
| Update Group Request Counters | 0 sent, 0 pending, 0 dropped |
| Comm Error Reauthenticator Counters | 0 topic connection drops detected |
| Agent Remote Scan Request Counters | 0 sent, 0 pending, 0 dropped |
| Agent State Change Counters | 0 sent, 0 pending, 0 dropped |
| Distributed Cache Publisher | sent: 0 bootstrap requests |
| Distributed Cache Subscriber | received: 0 activity messages, 0 activity events, 0 bootstrap messages, 0 bootstrap elements |
| Distributed Cache Contents | 'EndSystem' (0) |
| NAC Web Service Client | up, ready, since Fri May 25 16:29:09 MSK 2018 |
| NAC AAA Thread Counter | Thread[NAC AAA Server Request Processor (127.0.0.1 port:1300),7,NacAAARequestHandler Group](ThreadGroup: 9), Max: 8 @ Fri May 25 16:24:00 MSK 2018 |
| NAC ACCT Thread Counter | Thread[NAC ACCT Server Request Processor (127.0.0.1 port:1302),7,NacACCTRequestHandler Group](ThreadGroup: 5), Max: 4 @ Fri May 25 16:24:00 MSK 2018 |
| Last Request Processed | Thu Jan 01 03:00:00 MSK 1970 |
| Throttled Radius Requests | 0 |
| NetBIOS Requests | 0 |

#-------------------------------------------------------------------------------
# NAC Thread Pool Details
#-------------------------------------------------------------------------------

| Thread Name | Active Count | Pool Size | Queue Size | Max Queue Size | Queue Limit Reached | Throttled Tasks | Tasks Completed |
| ________________________________________________________ | ____________ | _________ | __________ | ______________ | ___________________ | _______________ | _______________ |
| Assessment Controller Thread Pool | 0 | 10 | 0 | 12000 | | 0 | 0 |
| EnforceHandler - Notify Listeners Thread Pool | 0 | 1 | 0 | 12000 | | 0 | 18 |
| EnforceHandler - Off Thread Notify Listeners Thread Pool | 0 | 1 | 0 | 12000 | | 0 | 3 |
| Initialize Switch Thread Thread Pool | 0 | 20 | 0 | 12000 | | 0 | 1 |
| NAC 2 NAC Message Handler Thread Pool | 0 | 1 | 0 | 10000 | | 0 | 74 |
| NAC Manager Config Message Handler Thread Pool | 0 | 1 | 0 | 12000 | | 0 | 0 |
| NAC Manager Status Message Handler Thread Pool | 0 | 1 | 0 | 12000 | | 0 | 0 |
| NAC Status Request Executor Thread Pool | 0 | 1 | 0 | 12000 | | 0 | 0 |
| NacCaptivePortalMainAction - Task Thread Pool | 0 | 10 | 0 | 12000 | | 0 | 0 |
| NetBIOS Request Manager Thread Pool | 0 | 5 | 0 | 500 | | 0 | 0 |
| RADIUS Session Deactivate Queue Thread Pool | 0 | 1 | 0 | 12000 | | 0 | 0 |
| SNMP Manager Refresh Child Thread Pool | 0 | 1 | 0 | 12000 | | 0 | 0 |
| SNMP Manager Refresh Parent Thread Pool | 0 | 1 | 0 | 12000 | | 0 | 0 |
| Switch Configuration Thread Pool | 0 | 1 | 0 | 10000 | | 0 | 2 |
| Switch Configuration Scheduled Thread Pool | 0 | 1 | 0 | 10000 | | 0 | 1 |
| Switch Configuration Task Thread Pool | 0 | 10 | 0 | 10000 | | 0 | 1 |
| TopicSubPub MessageMaker Thread Pool | 0 | 2 | 0 | 12000 | | 0 | 0 |

#-------------------------------------------------------------------------------
# NetSight Server Name Resolution
#-------------------------------------------------------------------------------

Resolving NetSight Server Name: NetSight
Server: 192.168.1.2
Address: 192.168.1.2#53

Name: NetSight.kafedra.local
Address: 192.168.1.201

#-------------------------------------------------------------------------------
# NAC Server Name Resolution
#-------------------------------------------------------------------------------

Resolving NAC Server Name: nac.kafedra.local
Server: 192.168.1.2
Address: 192.168.1.2#53

Name: nac.kafedra.local
Address: 192.168.1.200

#-------------------------------------------------------------------------------
# Communications Diagnostics
#-------------------------------------------------------------------------------

NAC to NetSight WebServices: SUCCESS.
NetSight to NAC Appliance WebServices: SUCCESS.
JMS Topic Connection: DOWN.
NetSight Server IP: 192.168.1.201
DNS Server IP: 192.168.1.2
NAC Domain Name: kafedra.local
Reverse DNS Lookup Timeout: 10
Reverse DNS Lookup of NAC Address: netsight (< 1 sec)
NAC Registration and Remediation IP: 192.168.1.200
NAC Hostname DNS Resolution: 192.168.1.200

RE: NAC appliance is red in console, but green in XMC

Ryan_Yacobucci — Mon, 28 May 2018 19:05:00 GMT

Hello,

First i'd make sure that you're not seeing an active alarm. Can you make sure and clear the alarms on the appliance?

Next, in NetSight Console right click on the NAC appliance and chose "MIB tools".

Does the bottom bar on the MIB tools window show an error like "Authentication failed"?

Check and make sure that the NAC has the correct profile, with the correct authentication/privacy parameters.

It needs to be set to auth/priv, and the credentials can be checked/reconfigured by running the "nacconfig" command on the NAC appliance itself.

Thanks
-Ryan

RE: NAC appliance is red in console, but green in XMC

Ilya_Semenov — Mon, 28 May 2018 19:05:00 GMT

Hello, Ryan,

I've done a new installation. Everything is 8.1. Now under ESXi 6.0. All VMs are in the same subnet, no firewall between them.

Both NACs are green in XMC, but red in NAC console. They all ping each other by IPs and hostnames.

I can see NAC's MIB Profiles on Netsight Console, no errors.

I use default snmp_v3_profile, everything is AuthPriv, and I left exactly these settings during appliance installation:

Nacstatus on both appliances says:

root@nac1.spbstu.ru:~$ nacstatus

#-------------------------------------------------------------------------------
# NAC Status
#-------------------------------------------------------------------------------

NAC Device Type: iav
NAC Device Version: 8.1.2.60
NAC OS Version: Ubuntu 14.04lts (64bit)
Management IP: 192.168.245.184

#-------------------------------------------------------------------------------
# Configuration Details
#-------------------------------------------------------------------------------

| EAC Engine Information | Access Control Engine - IA-V v.8.1.2.60 |
| License Status | No License - this appliance will not operate without a valid license |
| Hypervisor | VMWare ESX (0xEA580) |
| Extreme Access Control(EAC) Engine IP | 192.168.245.185 |
| Extreme Management Server IP Address | 192.168.245.184 |
| EAC Server Status | up, ready since Fri Jun 08 14:42:04 MSK 2018 |
| EAC Up Time (HH:MM:SS.mmmm) | 00:22:42.158 |

#-------------------------------------------------------------------------------
# Resource Details
#-------------------------------------------------------------------------------

| CPU Usage | User=18.59%, System=1.10%, Niced=0.00%, Idle=80.31%, Total=19.69% |
| Memory Usage | Used=10.44%, Free=89.56%, Total=11.73 GB |
| Swap Space | Used=0.00%, Free=100.00%, Total=11.73 GB |
| EAC Process | Heap=75.39%, Non-Heap=24.61%, Total=268.72 MB |
| Available Space | Path=/, Free-Space=22Gb, Total-Space=27Gb |

#-------------------------------------------------------------------------------
# Status Details
#-------------------------------------------------------------------------------

| Statistic | Current | Maximum | Total | Max Reached |
| _________________________________ | _______ | _______ | _____ | _____________ |
| Authentication Requests | 0/min | 0/min | 0 | Not Available |
| Authentication Successes | 0/min | 0/min | 0 | Not Available |
| Authentication Failures | 0/min | 0/min | 0 | Not Available |
| Radius Challenges | 0/min | 0/min | 0 | Not Available |
| Invalid Authentication Requests | 0/min | 0/min | 0 | Not Available |
| Duplicate Authentication Requests | 0/min | 0/min | 0 | Not Available |
| Malformed Authentication Requests | 0/min | 0/min | 0 | Not Available |
| Bad Authentication Requests | 0/min | 0/min | 0 | Not Available |
| Dropped Radius Packets | 0/min | 0/min | 0 | Not Available |
| Unknown Radius Types | 0/min | 0/min | 0 | Not Available |
| Assessment Requests | 0/min | 0/min | 0 | Not Available |
| Captive Portal Requests | 0/min | 0/min | 0 | Not Available |
| Contact Lost Switches | 0 | 0 | | Not Available |
| IP Resolution Failures | 0/min | 0/min | 0 | Not Available |
| IP Resolution Timeouts | 0/min | 0/min | 0 | Not Available |
| Connected Agents | 0 | 0 | | Not Available |
| End-System Events | 0/min | 0/min | 0 | Not Available |
| End-Systems One Day Count | 0 | 0 | | Not Available |
| End-Systems Current Count | 0 | 0 | | Not Available |

| EAC Manager Connection | down, not ready, since Thu Jan 01 03:00:00 MSK 1970 |
| General Message Counters | 0 sent, 9 dropped |
| Event Message Status | normal mode, since Fri Jun 08 14:42:08 MSK 2018 |
| Event Message Counters | 0 sent, 0 pending, 0 dropped |
| Health Result Message Status | normal mode, since Fri Jun 08 14:42:08 MSK 2018 |
| Health Result Message Counters | 0 sent, 0 pending, 0 dropped |
| EAC-to-EAC Message Status | merging mode, since Fri Jun 08 14:42:08 MSK 2018 |
| EAC-to-EAC Mergable Message Counters | 0 sent, 1 pending, 0 dropped |
| EAC-to-EAC Normal Message Counters | 0 sent, 1 pending, 0 dropped |
| Update Group Request Counters | 0 sent, 0 pending, 0 dropped |
| Comm Error Reauthenticator Counters | 0 topic connection drops detected |
| Agent Remote Scan Request Counters | 0 sent, 0 pending, 0 dropped |
| Agent State Change Counters | 0 sent, 0 pending, 0 dropped |
| EAC Web Service Client | down, not ready, since Fri Jun 08 14:42:13 MSK 2018 |
| EAC AAA Thread Counter | Thread[EAC AAA Server Request Processor (127.0.0.1 port:1300),7,EacAAARequestHandler Group](ThreadGroup: 2), Max: 1 @ Fri Jun 08 14:42:22 MSK 2018 |
| EAC ACCT Thread Counter | Thread[EAC ACCT Server Request Processor (127.0.0.1 port:1302),7,EacACCTRequestHandler Group](ThreadGroup: 1), Max: 0 @ Thu Jan 01 03:00:00 MSK 1970 |
| Last Request Processed | Thu Jan 01 03:00:00 MSK 1970 |
| Throttled Radius Requests | 0 |
| NetBIOS Requests | 0 |

#-------------------------------------------------------------------------------
# EAC Thread Pool Details
#-------------------------------------------------------------------------------

| Thread Name | Active Count | Pool Size | Queue Size | Max Queue Size | Queue Limit Reached | Throttled Tasks | Tasks Completed |
| ________________________________________________________ | ____________ | _________ | __________ | ______________ | ___________________ | _______________ | _______________ |
| Assessment Controller Thread Pool | 0 | 10 | 0 | 6000 | | 0 | 0 |
| EAC 2 EAC Message Handler Thread Pool | 0 | 1 | 0 | 10000 | | 0 | 31 |
| EAC Manager Config Message Handler Thread Pool | 0 | 1 | 0 | 6000 | | 0 | 0 |
| EAC Manager Status Message Handler Thread Pool | 0 | 1 | 0 | 6000 | | 0 | 0 |
| EAC Status Request Executor Thread Pool | 0 | 1 | 0 | 6000 | | 0 | 0 |
| EnforceHandler - Off Thread Notify Listeners Thread Pool | 0 | 1 | 0 | 6000 | | 0 | 1 |
| Initialize Switch Thread Thread Pool | 0 | 20 | 0 | 6000 | | 0 | 0 |
| NetBIOS Request Manager Thread Pool | 0 | 5 | 0 | 500 | | 0 | 0 |
| SNMP Manager Refresh Child Thread Pool | 0 | 1 | 0 | 6000 | | 0 | 0 |
| SNMP Manager Refresh Parent Thread Pool | 0 | 1 | 0 | 6000 | | 0 | 0 |
| Switch Configuration Scheduled Thread Pool | 0 | 1 | 0 | 10000 | | 0 | 1 |
| TopicSubPub MessageMaker Thread Pool | 0 | 2 | 0 | 6000 | | 0 | 0 |

#-------------------------------------------------------------------------------
# Startup End-System Auth Count Information
#-------------------------------------------------------------------------------

Current End-System count from last day at startup is: 0

Current active (not disconnected) End-System count at startup is: 0

Totals State - Accept: 0, Reject: 0, Scan: 0, Quarantine: 0, Error: 0, Disconnected: 0

Totals ConnectedState - Active: 0, Active with Highest Precedence: 0, Disconnected: 0, Unknown: 0

#-------------------------------------------------------------------------------
# NetSight Server Name Resolution
#-------------------------------------------------------------------------------

#-------------------------------------------------------------------------------
# NAC Server Name Resolution
#-------------------------------------------------------------------------------

Resolving NAC Server Name: nac1.spbstu.ru
Server: 194.190.225.225
Address: 194.190.225.225#53

Name: nac1.spbstu.ru
Address: 192.168.245.185

#-------------------------------------------------------------------------------
# Communications Diagnostics
#-------------------------------------------------------------------------------

NAC to NetSight WebServices: FAILURE.
NetSight to NAC WebServices: UNABLE TO TEST.
JMS Topic Connection: DOWN.
NetSight Server IP: 192.168.245.184
DNS Server IP: 194.190.225.225
NAC Domain Name: spbstu.ru
Reverse DNS Lookup Timeout: 10
Reverse DNS Lookup of NAC Address: xmc.spbstu.ru (< 1 sec)
NAC Registration and Remediation IP: 192.168.245.185
NAC Hostname DNS Resolution: 192.168.245.185

#-------------------------------------------------------------------------------
# Appliance License and Capacity Diagnostics
#-------------------------------------------------------------------------------

NAC appliance is virtual.
Virtual NAC appliance is not licensed.
License Status: No License
License Data: null
Current End-System Capacity: 2000
Assessment Capable: False

#-------------------------------------------------------------------------------
# Distributed Cache Diagnostics
#-------------------------------------------------------------------------------

NAC appliance distributed cache is disabled.
Distributed Caches found: 0
Distributed Caches Counters:
+ Bootstrap Requests Sent: 0
+ Bootstrap Messages Received: 0
+ Bootstrap Elements Received: 0
+ Activity Messages Received: 0
+ Activity Events Received: 0

#-------------------------------------------------------------------------------
# Process Status
#-------------------------------------------------------------------------------

EAC Watchdog Process Check Success
Database process is running.
RADIUS Process is running.
EAC Process is running.

#-------------------------------------------------------------------------------
# Most Recent Errors from /var/log/syslog
#-------------------------------------------------------------------------------

Jun 8 14:21:44 nac1 kernel: [ 4.481702] EXT4-fs (dm-0): re-mounted. Opts: errors=remount-ro

#-------------------------------------------------------------------------------
# Most Recent Actions from /var/log/watchdog.log
#-------------------------------------------------------------------------------

2018-06-08 14:21:48,374 INFO [SyslogWriter] Watchdog Service is starting
2018-06-08 14:42:03,596 INFO [SyslogWriter] Watchdog Service is starting

#-------------------------------------------------------------------------------
# Most Recent Errors from /var/log/tag.log
#-------------------------------------------------------------------------------

#-------------------------------------------------------------------------------
# Most Recent Errors from /var/log/radius/radius.log
#-------------------------------------------------------------------------------

Thu Jun 7 15:52:20 2018 : Error: [etsnac connection_mgr] Failed to connect to server 127.0.0.1 on port: 1300 with error: Connection refused(111)
Thu Jun 7 16:07:22 2018 : Error: [etsnac connection_mgr] Failed to connect to server 127.0.0.1 on port: 1300 with error: Connection refused(111)
Fri Jun 8 14:21:47 2018 : Error: [etsnac connection_mgr] Failed to connect to server 127.0.0.1 on port: 1300 with error: Connection refused(111)
Fri Jun 8 14:42:02 2018 : Error: [etsnac connection_mgr] Failed to connect to server 127.0.0.1 on port: 1300 with error: Connection refused(111)

#-------------------------------------------------------------------------------
# ProxyRedirect status
#-------------------------------------------------------------------------------

ProxyRedirector threads running: 0

#-------------------------------------------------------------------------------
# Squid Status
#-------------------------------------------------------------------------------

ERROR: Cannot connect to 127.0.0.1:3128

#-------------------------------------------------------------------------------
# NetSight server status
#-------------------------------------------------------------------------------

Checking Status of Access Control Engine, RADIUS, Proxy & Agentless Assessment Server:
Access Control Engine Proxy is NOT running...
Access Control Engine Server is running with PID: 4659
Access Control Engine RADIUS Server is running with PID: 4633
Agentless Assessment Server is running with PID: 4561

Run '/sbin/nacctl restart'.

#-------------------------------------------------------------------------------
# Hostname Information
#-------------------------------------------------------------------------------

Hostname: nac1.spbstu.ru
################################################################################
## hosts - hosts - local host configuration file
##
## WARNING: This file is automatically generated on every enforce.
## This file is made from the following templates. Any modifications
## should be made to the template files, not this file.
##
## templates/hosts.tpl
##
################################################################################

#
# hosts

RE: NAC appliance is red in console, but green in XMC

Ryan_Yacobucci — Mon, 28 May 2018 19:05:00 GMT

Hello,

I would advise creation of a case for further investigation.

NAC to NetSight WebServices: FAILURE.
NetSight to NAC WebServices: UNABLE TO TEST.
JMS Topic Connection: DOWN.

Thanks
-Ryan

RE: NAC appliance is red in console, but green in XMC

PeterK — Tue, 14 Aug 2018 11:20:00 GMT

Hi Ilya,

have you solved the issue?
I'm currently having same behavior at one of my customers.

RE: NAC appliance is red in console, but green in XMC

Ilya_Semenov1 — Tue, 14 Aug 2018 11:20:00 GMT

Eventually, I've made a clean install of NAC. That's solved the issue. Time sync between EWC and NAC is a very important thing also - you should setup same NTP settings everywhere.