topic RE: Impossible to get output of remote ssh commands via cron in ExtremeSwitching (EXOS/Switch Engine)

Impossible to get output of remote ssh commands via cron

Iván_García_Díe — Fri, 13 May 2016 18:08:00 GMT

Hello,

I have made a bash script to get ipstats of several devices, and i have a problem with my X450a-48t, the script works fine if i execute it manually, but in crontab no output, i think because in cron no tty is allocated.

Example script:

#!/bin/bash

ssh -o "StrictHostKeyChecking no" admin@10.26.196.189 "show ipstats"

i have checked ssh with -t and -tt option, but doesn't work.

¿Any idea how to solve it? ¿another possibility to get this stats remotely?

Regards,

Ivan

RE: Impossible to get output of remote ssh commands via cron

Erik_Auerswald — Fri, 13 May 2016 19:55:00 GMT

Hi,

the script started by cron probably has no SSH agent available. You could try to specify the identity (key) file to use with the option -i.
ssh -i /path/to/key_file -o "StrictHostKeyChecking no" admin@10.26.196.189 "show ipstats"Regards,
Erik

RE: Impossible to get output of remote ssh commands via cron

Iván_García_Díe — Fri, 13 May 2016 19:55:00 GMT

Hi Erik,

No difference with -i option. There is no problem with authentication. I share debug output, you can see "Authentication succeeded". The problem i think is this line "debug2: channel 0: read failed"

OpenSSH_5.1p1, OpenSSL 0.9.8j-fips 07 Jan 2009
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug1: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to 10.26.196.189 [10.26.196.189] port 22.
debug1: Connection established.
debug1: permanently_set_uid: 0/0
debug2: key_type_from_name: unknown key type '-----BEGIN'
debug2: key_type_from_name: unknown key type '-----END'
debug1: identity file /root/.ssh/id_rsa type 1
debug2: key_type_from_name: unknown key type '-----BEGIN'
debug2: key_type_from_name: unknown key type '-----END'
debug1: identity file /root/.ssh/id_dsa type 2
debug1: Remote protocol version 2.0, remote software version 4.1.2 SSH Secure Shell Toolkit
debug1: no match: 4.1.2 SSH Secure Shell Toolkit
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.1
debug2: fd 3 setting O_NONBLOCK
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes256-ctr,arcfour256,arcfour,aes128-cbc,aes256-cbc
debug2: kex_parse_kexinit: aes128-ctr,aes256-ctr,arcfour256,arcfour,aes128-cbc,aes256-cbc
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: kex_parse_kexinit: diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-dss
debug2: kex_parse_kexinit: aes256-cbc,aes192-cbc,aes128-cbc,twofish256-cbc,twofish-cbc,twofish192-cbc,twofish128-cbc,blowfish-cbc,3des-cbc
debug2: kex_parse_kexinit: aes256-cbc,aes192-cbc,aes128-cbc,twofish256-cbc,twofish-cbc,twofish192-cbc,twofish128-cbc,blowfish-cbc,3des-cbc
debug2: kex_parse_kexinit: hmac-sha1,hmac-md5,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-sha1,hmac-md5,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none
debug2: kex_parse_kexinit: none
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: mac_setup: found hmac-md5
debug1: kex: server->client aes128-cbc hmac-md5 none
debug2: mac_setup: found hmac-md5
debug1: kex: client->server aes128-cbc hmac-md5 none
debug2: dh_gen_key: priv key bits set: 123/256
debug2: bits set: 494/1024
debug1: sending SSH2_MSG_KEXDH_INIT
debug1: expecting SSH2_MSG_KEXDH_REPLY
debug1: Host '10.26.196.189' is known and matches the DSA host key.
debug1: Found key in /root/.ssh/known_hosts:4
debug2: bits set: 525/1024
debug1: ssh_dss_verify: signature correct
debug2: kex_derive_keys
debug2: set_newkeys: mode 1
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug2: set_newkeys: mode 0
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug2: key: /root/.ssh/id_rsa (0x7f2e3b150be0)
debug2: key: /root/.ssh/id_dsa (0x7f2e3b150b30)
debug1: Authentications that can continue: publickey,keyboard-interactive,password
debug1: Next authentication method: publickey
debug1: Offering public key: /root/.ssh/id_rsa
debug2: we sent a publickey packet, wait for reply
debug1: Server accepts key: pkalg ssh-rsa blen 277
debug2: input_userauth_pk_ok: fp 2a:f5:57:63:32:81:62:57:31:e9:de:46:7e:49:eb:ca
debug1: read PEM private key done: type RSA
debug1: Authentication succeeded (publickey).
debug2: fd 4 setting O_NONBLOCK
debug2: fd 5 setting O_NONBLOCK
debug2: fd 6 setting O_NONBLOCK
debug1: channel 0: new [client-session]
debug2: channel 0: send open
debug1: Entering interactive session.
debug2: callback start
debug2: client_session2_setup: id 0
debug1: Sending environment.
debug1: Sending command: show ipstats
debug2: channel 0: request exec confirm 1
debug2: fd 3 setting TCP_NODELAY
debug2: callback done
debug2: channel 0: open confirm rwindow 16384 rmax 32768
debug2: channel 0: read<=0 rfd 4 len 0
debug2: channel 0: read failed
debug2: channel 0: close_read
debug2: channel 0: input open -> drain
debug2: channel 0: ibuf empty
debug2: channel 0: send eof
debug2: channel 0: input drain -> closed
debug2: channel_input_confirm: type 99 id 0
debug2: exec request accepted on channel 0
debug2: channel 0: rcvd eof
debug2: channel 0: output open -> drain
debug2: channel 0: rcvd close
debug2: channel 0: obuf empty
debug2: channel 0: close_write
debug2: channel 0: output drain -> closed
debug2: channel 0: almost dead
debug2: channel 0: gc: notify user
debug2: channel 0: gc: user detached
debug2: channel 0: send close
debug2: channel 0: is dead
debug2: channel 0: garbage collecting
debug1: channel 0: free: client-session, nchannels 1
debug1: fd 0 clearing O_NONBLOCK
debug1: fd 1 clearing O_NONBLOCK
debug1: fd 2 clearing O_NONBLOCK
Transferred: sent 1960, received 2008 bytes, in 0.1 seconds
Bytes per second: sent 21000.3, received 21514.6
debug1: Exit status -1

RE: Impossible to get output of remote ssh commands via cron

Erik_Auerswald — Fri, 13 May 2016 19:55:00 GMT

Hello Ivan,

did you try to use two -t options?
ssh -o "StrictHostKeyChecking no" -t -t admin@10.26.196.189 "show ipstats"As I understand it more than one -t option is needed to really force pseudo-tty allocation.

Erik

Edit: Just re-read your first post, you wrote that you already tried -tt...

RE: Impossible to get output of remote ssh commands via cron

Erik_Auerswald — Tue, 17 May 2016 14:53:00 GMT

Hm, I have just tested this with an X460-G2, EXOS 21.1.1.4, OpenSSH_6.6.1p1, and ISC cron 3.0pl1-124ubuntu2 on Ubuntu 14.04. The original command works fine, both from the command line and from cron (crontab of an unprivileged user). I have tested both from inside a BASH script and the ssh command directly in the crontab entry. No "-t" options needed.

RE: Impossible to get output of remote ssh commands via cron

Iván_García_Díe — Tue, 17 May 2016 16:14:00 GMT

I dont understand why it fails in my case. And only with this device.

But finally, trying another ways i succeeded using expect. The output of the command is saved in the expect log.

Thanks

RE: Impossible to get output of remote ssh commands via cron

Erik_Auerswald — Tue, 17 May 2016 16:15:00 GMT

Getting closer...

I have tested with an X450e-24p with EXOS 15.3.5.2, there I see the reported problem. I do not receive the switch output if the command is started from cron or with STDIN redirected from /dev/null. This seems to be an issue with the older firmware. 😞

RE: Impossible to get output of remote ssh commands via cron

Bart_Wallace — Sat, 09 Jul 2016 01:51:00 GMT

I have the same problem using a perl script. We are using the script to grab the configuration on a nightly basis, then commit the files to git for version control, backup, archive, etc.

The script executes perfectly if I run it manually from command line. However, running the same script within cron and it fails. I have a user account using a ssh user-key added to the admin account on the switch.

Here's the command executing by the script:
"ssh -oStrictHostKeyChecking=no admin\@$xxrip upload configuration $tftpserver $xxrname.xsf" where $xxrip, $tftpserver and $xxrname are passed into the command from a hash.

I've tried adding the -t -t option but no change.

My error code from the perl script for this command is 255, both from running the script manually, or through cron - I'm guessing 255 is "all ok" as the script works just fine running manually. However, the configuration file is never uploaded to my tftp server when running the script through cron.

I contacted tech support and opened a case (#01203422), but they've passed me off with the following:
I wanted to follow up with you on this case. Unfortunately at this point there's nothing more we can do to troubleshoot this as this is an issue with KRON and not the switch itself. Have you tried to contact support for the scheduler?"

I'm looking for any help with this. I can share my perl script and git methods with anyone that is interested. Having a copy of the latest configuration(s) has saved me a few times.

RE: Impossible to get output of remote ssh commands via cron

Erik_Auerswald — Sat, 09 Jul 2016 20:08:00 GMT

Hello Bart,

which version of EXOS do you use? In my testing older firmware versions did not work, while newer did.

Did you try to run your script with /dev/null as STDIN from a terminal window? That resulted in the same problem as running the command from cron for me.
$ my_script HTH,
Erik

RE: Impossible to get output of remote ssh commands via cron

Bart_Wallace — Sat, 09 Jul 2016 20:08:00 GMT

Erik,

I'm using on all my switches:
Image : ExtremeXOS version 15.3.5.2 v1535b2-patch1-9 by release-manager
on Fri Dec 11 12:07:39 EST 2015
BootROM : 1.0.3.5

I get the same result as you do with running my_script
Which version are you running?

RE: Impossible to get output of remote ssh commands via cron

Erik_Auerswald — Sat, 09 Jul 2016 20:08:00 GMT

Hi Bart,

I am using the same version (the x450e switches I am using cannot run newer EXOS versions). When testing this with the EXOS-VM version 21.1.14, ssh commands work fine with STDIN redirected from /dev/null, so inbetween those EXOS version something was changed. It might be a side effect of the newer SSH code in 21.1.

To summarize the situation for EXOS 15.3:

SSH commands work fine from an interactive shell (e.g. terminal window)
SSH commands with STDIN redirected from /dev/null do not work from an interactive shell (neither with ssh -n nor ssh SSH commands from a script started via cron do not work (here STDIN is redirected from /dev/null by cron)
expect can be used to work around this issue (e.g. the original TCL based expect, or a Perl extpect module, or one for Python)

I have successfully tested a simple expect script with EXOS 15.3 today. Expect provides the input to SSH on STDIN.

With EXOS version 21.1, SSH commands work even with STDIN redirected from /dev/null, e.g. when used via cron.

Best regards,
Erik

RE: Impossible to get output of remote ssh commands via cron

Bart_Wallace — Sat, 09 Jul 2016 20:08:00 GMT

Erik,

Thanks for the info. I'll look into expect perl modules. Can you provide the simple script you tested today so I can verify results?

Bart

RE: Impossible to get output of remote ssh commands via cron

Erik_Auerswald — Sat, 09 Jul 2016 20:08:00 GMT

Bart,

I'm not in the office today, but I plan to add the script to this thread tomorrow.

Erik

RE: Impossible to get output of remote ssh commands via cron

Erik_Auerswald — Sat, 09 Jul 2016 20:08:00 GMT

Hi,

I have used the following simple script just to test the switch behavior with SSH via expect. The switches had my SSH key configured for admin access. The switch IP address needs to be adjusted, of course.
#! /usr/bin/expect -f spawn ssh admin@192.0.2.11 expect { {# } {send "show version\r"} } expect { {# } {send "exit\r"} }The expect script waits for the switch prompt (login w/o password handled via SSH), issues the command "show version", and logs off the switch after detecting the prompt again.

Br,
Erik