Receiving "*** IDLE TIMEOUT ***" when running commands over SSH via bastion/jump server

[AlfredForbrich]

Hi, 

I need to automate execution of some of the "show ***" command, i.e. "show clock", on the switches in my network, SLX-OS 20.5.1b, but need to ssh to the switch via a bastion/jump server. I'm using python paramiko module to ssh and automate the running of the scripts. Unfortunately, while I appear to be connecting to the remote site and can login, my command, i.e. "show clock" is never executed and I end up with the following output ( I've redacted site info with <text> )

Contacting: 1<slx ip>
['show clock Last login: Mon Nov 27 05:14:53 2023 from <bastion IP>\r\n', 'Unsuccessful attempts: 0 since last login.\r\n', '\r\n', 'SECURITY WARNING: The default password for at least\r\n', 'one default account (root, admin and user) have not been changed.\r\n', '\r\n', 'Welcome to the Extreme SLX-OS Software\r\n', 'admin connected from <bastion IP> using ssh on N170SLX-BLB\r\n', '\x1b[?7h<SLX Name># \r\n', '*** IDLE TIMEOUT ***\r\n']

I have searched the internet and found this similar issue and tried to use asyncio and asyncssh but I still end up with a failure.

https://github.com/ronf/asyncssh/issues/567

DEBUG:asyncio:Using selector: EpollSelector
<bastion IP> 22 <bastion user> <bastion password> <slx IP> 22 <slx user> <slx password>
DEBUG:asyncssh:Reading config from "/home/cna/.ssh/config"
DEBUG:asyncssh:Reading config from "/home/cna/.ssh/config.d/...."
INFO:asyncssh:Opening SSH connection to <bastion IP>, port 22
INFO:asyncssh:[conn=0] Connected to SSH server at <bastion IP>, port 22
INFO:asyncssh:[conn=0] Local address: 10.233.70.115, port 39820
INFO:asyncssh:[conn=0] Peer address: <bastion IP>, port 22
DEBUG:asyncssh:[conn=0] Sending version SSH-2.0-AsyncSSH_2.14.1
DEBUG:asyncssh:[conn=0] Received version SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.7
DEBUG:asyncssh:[conn=0] Requesting key exchange
DEBUG:asyncssh:[conn=0] Received key exchange request
DEBUG:asyncssh:[conn=0] Beginning key exchange
DEBUG:asyncssh:[conn=0] Completed key exchange
INFO:asyncssh:[conn=0] Beginning auth for user extreme
DEBUG:asyncssh:[conn=0] Trying public key auth with rsa-sha2-256 key
DEBUG:asyncssh:[conn=0] Signing request with rsa-sha2-256 key
INFO:asyncssh:[conn=0] Auth for user extreme succeeded
DEBUG:asyncssh:Reading config from "/home/..."
DEBUG:asyncssh:Reading config from "/home/....."
INFO:asyncssh:[conn=0] Opening SSH connection to <slx IP>, port 22 via SSH tunnel
INFO:asyncssh:[conn=0] Opening direct TCP connection to <slx IP>, port 22
INFO:asyncssh:[conn=0] Client address: dynamic port
DEBUG:asyncssh:[conn=0, chan=0] Set write buffer limits: low-water=16384, high-water=65536
DEBUG:asyncssh:[conn=0] Received unknown global request: hostkeys-00@openssh.com
INFO:asyncssh:[conn=1] Connected to SSH server at <slx IP>, port 22
INFO:asyncssh:[conn=1] Local address: 10.233.70.115, port 39820
INFO:asyncssh:[conn=1] Peer address: dynamic port
DEBUG:asyncssh:[conn=1] Sending version SSH-2.0-AsyncSSH_2.14.1
DEBUG:asyncssh:[conn=1] Received version SSH-2.0-OpenSSH_7.5 PKIX[10.2]
DEBUG:asyncssh:[conn=1] Requesting key exchange
DEBUG:asyncssh:[conn=1] Received key exchange request
DEBUG:asyncssh:[conn=1] Beginning key exchange
DEBUG:asyncssh:[conn=1] Completed key exchange
INFO:asyncssh:[conn=1] Beginning auth for user admin
DEBUG:asyncssh:[conn=1] Received authentication banner
DEBUG:asyncssh:[conn=1] Trying public key auth with rsa-sha2-256 key
DEBUG:asyncssh:[conn=1] Trying password auth
INFO:asyncssh:[conn=1] Auth for user admin succeeded
DEBUG:asyncssh:[conn=1, chan=0] Set write buffer limits: low-water=16384, high-water=65536
INFO:asyncssh:[conn=1, chan=0] Requesting new SSH session
DEBUG:asyncssh:[conn=1] Received unknown global request: hostkeys-00@openssh.com
INFO:asyncssh:[conn=1, chan=0] Command: show clock
Result 1 stdout in 0:00:00.070467: Last login: Mon Nov 27 14:14:25 2023 from 192.168.246.11
Unsuccessful attempts: 0 since last login.

INFO:asyncssh:[conn=1, chan=0] Closing channel
INFO:asyncssh:[conn=1, chan=0] Received exit status 0
INFO:asyncssh:[conn=1, chan=0] Received channel close
INFO:asyncssh:[conn=1, chan=0] Channel closed
INFO:asyncssh:[conn=1] Closing connection
INFO:asyncssh:[conn=1] Sending disconnect: Disconnected by application (11)
INFO:asyncssh:[conn=0, chan=0] Aborting channel
INFO:asyncssh:[conn=1] Connection closed
INFO:asyncssh:[conn=0] Closing connection
INFO:asyncssh:[conn=0, chan=0] Closing channel
INFO:asyncssh:[conn=0] Sending disconnect: Disconnected by application (11)
INFO:asyncssh:[conn=0] Connection closed
INFO:asyncssh:[conn=0, chan=0] Closing channel due to connection close
INFO:asyncssh:[conn=0, chan=0] Channel closed

Note that I'm able to ssh to the bastion and through the bastion to the switch and run commands but this is not possible in the production environment.

[AlfredForbrich · Accepted solution]

It appears that I can work around my issue using the paramiko invoke_shell() function

proxy_jump_command='ssh -W {HOST}:{PORT} <bastion ssh info>'.format(HOST=slxAddress,PORT='22')
proxy=paramiko.ProxyCommand(proxy_jump_command)

try:
    ssh_client.connect(slxAddress, username='<slx user>', password='<slx user password>', sock=proxy )

except Exception as e:
    print("exception: {}".format(e))

if ssh_client.get_transport() is not None:
     ssh_client.get_transport().is_active()
     if ( debug 😞 print("connected: {}".format(ssh_client.get_transport().is_active()))
remote_conn = ssh_client.invoke_shell()
remote_conn.send('\r')
output = remote_conn.recv(65535)
time.sleep(0.5)

for cmd in slxcli_command_list :
     cmd = cmd + "\r"
     try:
           remote_conn.send(cmd)
           time.sleep(5.0)
          output = remote_conn.recv(65535)
     except Exception as e:
          print("Exception: {}".format(e))
    print("output: {}".format(output))
ssh_client.close()