NAC appliance is red in console, but green in XMC

[Ilya_Semenov]

Hello, team,

after reboot NAC is red in Console, but green in XMC. As result, nothing works.

I've read:

https://extremeportal.force.com/ExtrArticleDetail?an=000077969
https://extremeportal.force.com/ExtrArticleDetail?an=000078011
https://extremeportal.force.com/ExtrArticleDetail?an=000063624

Nothing helped me. Curious, that nacstatus says that everything is OK.

root@nac.kafedra.local:/var/log$ nacstatus

#-------------------------------------------------------------------------------
# NAC Status
#-------------------------------------------------------------------------------

NAC Device Type: iav
NAC Device Version: 7.1.1.9
NAC OS Version: Ubuntu 12.04lts (64bit)
Management IP: 192.168.1.201

#-------------------------------------------------------------------------------
# Configuration Details
#-------------------------------------------------------------------------------

| NAC Engine Information | Access Control Engine - NETSIGHTEVAL v.7.1.1.9 |
| License Status | Valid License [netsighteval] (Evaluation period expires in 64 days) |
| Hypervisor | Microsoft Hyper-V |
| NAC Engine IP | 192.168.1.200 |
| NetSight Server IP Address | 192.168.1.201 |
| NAC Server Status | up, ready since Fri May 25 16:17:58 MSK 2018 |
| NAC Up Time (HH:MM:SS.mmmm) | 00:26:19.143 |

#-------------------------------------------------------------------------------
# Resource Details
#-------------------------------------------------------------------------------

| CPU Usage | User=4.93%, System=1.75%, Niced=0.00%, Idle=93.32%, Total=6.68% |
| Memory Usage | Used=83.96%, Free=16.04%, Total=7.78 GB |
| Swap Space | Used=0.00%, Free=100.00%, Total=7.78 GB |
| NAC Process | Heap=82.89%, Non-Heap=17.11%, Total=426.4 MB |
| Available Space | Path=/, Free-Space=30Gb, Total-Space=35Gb |

#-------------------------------------------------------------------------------
# Status Details
#-------------------------------------------------------------------------------

| Statistic | Current | Maximum | Total | Max Reached |
| _________________________________ | _______ | _______ | _____ | ____________________________ |
| Authentication Requests | 0/min | 0/min | 0 | Not Available |
| Authentication Successes | 0/min | 0/min | 0 | Not Available |
| Authentication Failures | 0/min | 0/min | 0 | Not Available |
| Radius Challenges | 0/min | 0/min | 0 | Not Available |
| Invalid Authentication Requests | 0/min | 0/min | 0 | Not Available |
| Duplicate Authentication Requests | 0/min | 0/min | 0 | Not Available |
| Malformed Authentication Requests | 0/min | 0/min | 0 | Not Available |
| Bad Authentication Requests | 0/min | 0/min | 0 | Not Available |
| Dropped Radius Packets | 0/min | 0/min | 0 | Not Available |
| Unknown Radius Types | 0/min | 0/min | 0 | Not Available |
| Assessment Requests | 0/min | 0/min | 0 | Not Available |
| Captive Portal Requests | 0/min | 15/min | 32 | Fri May 25 16:21:04 MSK 2018 |
| Contact Lost Switches | 0 | 0 | | Not Available |
| IP Resolution Failures | 0/min | 0/min | 0 | Not Available |
| IP Resolution Timeouts | 0/min | 0/min | 0 | Not Available |
| Connected Agents | 0 | 0 | | Not Available |
| End-System Events | 0/min | 0/min | 0 | Not Available |
| End-Systems One Day Count | 8 | 8 | | Fri May 25 16:18:04 MSK 2018 |
| End-Systems Current Count | 8 | 8 | | Fri May 25 16:18:04 MSK 2018 |

| NAC Manager Connection | down, ready, since Thu Jan 01 03:00:00 MSK 1970 |
| General Message Counters | 0 sent, 12 dropped |
| Event Message Status | normal mode, since Fri May 25 16:18:01 MSK 2018 |
| Event Message Counters | 0 sent, 0 pending, 0 dropped |
| Health Result Message Status | normal mode, since Fri May 25 16:18:01 MSK 2018 |
| Health Result Message Counters | 0 sent, 0 pending, 0 dropped |
| NAC-to-NAC Message Status | merging mode, since Fri May 25 16:18:01 MSK 2018 |
| NAC-to-NAC Mergable Message Counters | 0 sent, 2 pending, 0 dropped |
| NAC-to-NAC Normal Message Counters | 0 sent, 2 pending, 0 dropped |
| Update Group Request Counters | 0 sent, 0 pending, 0 dropped |
| Comm Error Reauthenticator Counters | 0 topic connection drops detected |
| Agent Remote Scan Request Counters | 0 sent, 0 pending, 0 dropped |
| Agent State Change Counters | 0 sent, 0 pending, 0 dropped |
| Distributed Cache Publisher | sent: 0 bootstrap requests |
| Distributed Cache Subscriber | received: 0 activity messages, 0 activity events, 0 bootstrap messages, 0 bootstrap elements |
| Distributed Cache Contents | 'EndSystem' (0) |
| NAC Web Service Client | up, ready, since Fri May 25 16:29:09 MSK 2018 |
| NAC AAA Thread Counter | Thread[NAC AAA Server Request Processor (127.0.0.1 port:1300),7,NacAAARequestHandler Group](ThreadGroup: 9), Max: 8 @ Fri May 25 16:24:00 MSK 2018 |
| NAC ACCT Thread Counter | Thread[NAC ACCT Server Request Processor (127.0.0.1 port:1302),7,NacACCTRequestHandler Group](ThreadGroup: 5), Max: 4 @ Fri May 25 16:24:00 MSK 2018 |
| Last Request Processed | Thu Jan 01 03:00:00 MSK 1970 |
| Throttled Radius Requests | 0 |
| NetBIOS Requests | 0 |

#-------------------------------------------------------------------------------
# NAC Thread Pool Details
#-------------------------------------------------------------------------------

| Thread Name | Active Count | Pool Size | Queue Size | Max Queue Size | Queue Limit Reached | Throttled Tasks | Tasks Completed |
| ________________________________________________________ | ____________ | _________ | __________ | ______________ | ___________________ | _______________ | _______________ |
| Assessment Controller Thread Pool | 0 | 10 | 0 | 12000 | | 0 | 0 |
| EnforceHandler - Notify Listeners Thread Pool | 0 | 1 | 0 | 12000 | | 0 | 18 |
| EnforceHandler - Off Thread Notify Listeners Thread Pool | 0 | 1 | 0 | 12000 | | 0 | 3 |
| Initialize Switch Thread Thread Pool | 0 | 20 | 0 | 12000 | | 0 | 1 |
| NAC 2 NAC Message Handler Thread Pool | 0 | 1 | 0 | 10000 | | 0 | 74 |
| NAC Manager Config Message Handler Thread Pool | 0 | 1 | 0 | 12000 | | 0 | 0 |
| NAC Manager Status Message Handler Thread Pool | 0 | 1 | 0 | 12000 | | 0 | 0 |
| NAC Status Request Executor Thread Pool | 0 | 1 | 0 | 12000 | | 0 | 0 |
| NacCaptivePortalMainAction - Task Thread Pool | 0 | 10 | 0 | 12000 | | 0 | 0 |
| NetBIOS Request Manager Thread Pool | 0 | 5 | 0 | 500 | | 0 | 0 |
| RADIUS Session Deactivate Queue Thread Pool | 0 | 1 | 0 | 12000 | | 0 | 0 |
| SNMP Manager Refresh Child Thread Pool | 0 | 1 | 0 | 12000 | | 0 | 0 |
| SNMP Manager Refresh Parent Thread Pool | 0 | 1 | 0 | 12000 | | 0 | 0 |
| Switch Configuration Thread Pool | 0 | 1 | 0 | 10000 | | 0 | 2 |
| Switch Configuration Scheduled Thread Pool | 0 | 1 | 0 | 10000 | | 0 | 1 |
| Switch Configuration Task Thread Pool | 0 | 10 | 0 | 10000 | | 0 | 1 |
| TopicSubPub MessageMaker Thread Pool | 0 | 2 | 0 | 12000 | | 0 | 0 |

#-------------------------------------------------------------------------------
# NetSight Server Name Resolution
#-------------------------------------------------------------------------------

Resolving NetSight Server Name: NetSight
Server: 192.168.1.2
Address: 192.168.1.2#53

Name: NetSight.kafedra.local
Address: 192.168.1.201

#-------------------------------------------------------------------------------
# NAC Server Name Resolution
#-------------------------------------------------------------------------------

Resolving NAC Server Name: nac.kafedra.local
Server: 192.168.1.2
Address: 192.168.1.2#53

Name: nac.kafedra.local
Address: 192.168.1.200

#-------------------------------------------------------------------------------
# Communications Diagnostics
#-------------------------------------------------------------------------------

NAC to NetSight WebServices: SUCCESS.
NetSight to NAC Appliance WebServices: SUCCESS.
JMS Topic Connection: DOWN.
NetSight Server IP: 192.168.1.201
DNS Server IP: 192.168.1.2
NAC Domain Name: kafedra.local
Reverse DNS Lookup Timeout: 10
Reverse DNS Lookup of NAC Address: netsight (< 1 sec)
NAC Registration and Remediation IP: 192.168.1.200
NAC Hostname DNS Resolution: 192.168.1.200

[PeterK]

Hi Ilya,

have you solved the issue?
I'm currently having same behavior at one of my customers.

[Ilya_Semenov1]

Eventually, I've made a clean install of NAC. That's solved the issue. Time sync between EWC and NAC is a very important thing also - you should setup same NTP settings everywhere.

[Ryan_Yacobucci]

Hello,

First i'd make sure that you're not seeing an active alarm. Can you make sure and clear the alarms on the appliance?

Next, in NetSight Console right click on the NAC appliance and chose "MIB tools".

Does the bottom bar on the MIB tools window show an error like "Authentication failed"?

Check and make sure that the NAC has the correct profile, with the correct authentication/privacy parameters.

It needs to be set to auth/priv, and the credentials can be checked/reconfigured by running the "nacconfig" command on the NAC appliance itself.

Thanks
-Ryan

[Ryan_Yacobucci]

Hello,

I would advise creation of a case for further investigation.

NAC to NetSight WebServices: FAILURE.
NetSight to NAC WebServices: UNABLE TO TEST.
JMS Topic Connection: DOWN.

Thanks
-Ryan