We use one 10Gb port for the management (ESA2) and tag all vlans to the port. (most VNS are tunneled back to the controller) the port links directly into the core. same for the other controller.
The management subnet is a small /28 subnet with the other controller in it.
the EMC server is on another subnet but routed within the LAN.
There is a simple default route to the management subnet gateway for all traffic. Nothing too fancy here.
Primary controller hasn't had any issues. suspect the other with the issue actually has a problem but as we run all APs to the primary controller as local APs we don't really see any traffic on this problematic controller to determine if any other issues.
we are having the exact same problem with our 2 C5210 controllers, once my controllers report down they do not come back up even after rediscovering. However your description is more succinct than mine. We have tried all of the things you have with the same results.
What is your configuration in regards to the physical ports? Is the port that you use to manage the controller (allow management traffic) on the same subnet as the EMC?
In my configuration it is not on the same subnet but it is on a directly connected network. Other devices on that subnet have no problem connecting and talking to the EMC.
One suggestion we had from GTAC was to reduce the number of "physical" ports to 1 on the controller. I deleted all the other physical ports to just the 1 I am using for the APs and management. This seemed to work for a while but then the controllers dropped off again and I could not re-establish communications.
I have connected the management port directly to the same subnet as the EMC. This has provided a stable connection to the EMC console but device collection statistics for OneView does not appear to work. The controllers are always saying down on the OneView dashboards however we do have our AP alarms back which is helpful.
I do not like this work around as it is not working completely and I do not like using the management port as the documentation says not to use it.
I have a case open with GTAC and my own thread on the hub (Is it possible to direct SNMP traffic to a specific physical port on a EWC C5210). I will follow your topic to see if you have any updates and let you know if anything is found from my own inquiries.