There's an inherent flaw in out-of-band management for network infrastructure. If your network infrastructure fails and you don't have a path back, then the out-of-band port is effectively down, too. So, you need to make sure that you have alternate paths preferrably with redundant hardware as well.
This idea works well on servers because they are not network devices, and rely on the network's resiliency.
I will echo Ronald's reply, though. You are looking for a terminal server. Just know it may not do much good if you suffer a failure on your uplink, or the switch the terminal server is connected to.