Problem with backup config over SSH

Mikhail_Ivanov — Tue, 01 Sep 2015 14:23:00 GMT

Problem appear when make automated config-backups over SSH-protocol with key-based authentication.

RE: Problem with backup config over SSH

Mikhail_Ivanov — Tue, 01 Sep 2015 14:49:00 GMT

How to reproduce:
I have Extreme Summit x460-48x switch with XOS 15.3.3.5 (patch1-4) with ssh-module.
I configured user 'backuper' for key-based auth in ssh:

----------------------------------
create account admin backuper encrypted ""
create sshd2 user-key backuper_ssh_pubkey AAAA
create sshd2 user-key backuper2_ssh_pubkey AAAA
configure sshd2 user-key backuper_ssh_pubkey add user backuper
configure sshd2 user-key backuper2_ssh_pubkey add user backuper
----------------------------------

User 'backuper' contain two public-keys - each key for each backup-server.
I make backups from linux-server (Fedora 21) with this command:

----------------------------------
ssh 10.44.0.15 "show configuration" > ex460-48x_1.conf 2>/dev/null
----------------------------------

And have strange behavior - when i running this command in bash/sh shell manually, command work correctly, and configuration-dump stored in ex460-48x_1.conf .

But when i running this command from bash-script, executed by crontab every night, the result file contain only 2 bytes 0x0D 0x0A.

I turn on debug-log for ssh-client in my backup-script:

----------------------------------
ssh -v -v -v 10.44.0.15 "show configuration" 1> ex460-48x_gate.conf 2> ex460-48x_gate.log
----------------------------------

Detailed SSH-log, writed by cron:

----------------------------------
OpenSSH_6.6.1, OpenSSL 1.0.1k-fips 8 Jan 2015
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 51: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to 10.44.0.15.225 [10.44.0.15] port 22.
debug1: Connection established.
debug3: Incorrect RSA1 identifier
debug3: Could not load "/home/backuper/.ssh/id_rsa" as a RSA1 public key
debug1: identity file /home/backuper/.ssh/id_rsa type 1
debug1: identity file /home/backuper/.ssh/id_rsa-cert type -1
debug1: identity file /home/backuper/.ssh/id_dsa type -1
debug1: identity file /home/backuper/.ssh/id_dsa-cert type -1
debug1: identity file /home/backuper/.ssh/id_ecdsa type -1
debug1: identity file /home/backuper/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/backuper/.ssh/id_ed25519 type -1
debug1: identity file /home/backuper/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.6.1
debug1: Remote protocol version 2.0, remote software version 4.1.2 SSH Secure Shell Toolkit
debug1: no match: 4.1.2 SSH Secure Shell Toolkit
debug2: fd 3 setting O_NONBLOCK
debug3: load_hostkeys: loading entries for host "10.44.0.15" from file "/home/backuper/.ssh/known_hosts"
debug3: load_hostkeys: found key type DSA in file /home/backuper/.ssh/known_hosts:7
debug3: load_hostkeys: loaded 1 keys
debug3: order_hostkeyalgs: prefer hostkeyalgs: ssh-dss-cert-v01@openssh.com,ssh-dss-cert-v00@openssh.com,ssh-dss
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-dss-cert-v01@openssh.com,ssh-dss-cert-v00@openssh.com,ssh-dss,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,ssh-rsa
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: hmac-md5-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-sha1,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-sha1,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: kex_parse_kexinit: diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-dss
debug2: kex_parse_kexinit: aes256-cbc,aes192-cbc,aes128-cbc,twofish256-cbc,twofish-cbc,twofish192-cbc,twofish128-cbc,blowfish-cbc,3des-cbc
debug2: kex_parse_kexinit: aes256-cbc,aes192-cbc,aes128-cbc,twofish256-cbc,twofish-cbc,twofish192-cbc,twofish128-cbc,blowfish-cbc,3des-cbc
debug2: kex_parse_kexinit: hmac-sha1,hmac-md5,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-sha1,hmac-md5,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none
debug2: kex_parse_kexinit: none
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: mac_setup: setup hmac-md5
debug1: kex: server->client aes128-cbc hmac-md5 none
debug2: mac_setup: setup hmac-md5
debug1: kex: client->server aes128-cbc hmac-md5 none
debug1: kex: diffie-hellman-group1-sha1 need=16 dh_need=16
debug1: kex: diffie-hellman-group1-sha1 need=16 dh_need=16
debug2: bits set: 526/1024
debug1: sending SSH2_MSG_KEXDH_INIT
debug1: expecting SSH2_MSG_KEXDH_REPLY
debug1: Server host key: DSA 9a:a5:eb:a2:4b:22:d8:15:ab:f0:c7:da:a3:f1:3a:56
debug3: load_hostkeys: loading entries for host "10.44.0.15" from file "/home/backuper/.ssh/known_hosts"
debug3: load_hostkeys: found key type DSA in file /home/backuper/.ssh/known_hosts:7
debug3: load_hostkeys: loaded 1 keys
debug1: Host '10.44.0.15' is known and matches the DSA host key.
debug1: Found key in /home/backuper/.ssh/known_hosts:7
debug2: bits set: 524/1024
debug1: ssh_dss_verify: signature correct
debug2: kex_derive_keys
debug2: set_newkeys: mode 1
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug2: set_newkeys: mode 0
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug2: key: /home/backuper/.ssh/id_rsa (0x55c49f0af290),
debug2: key: /home/backuper/.ssh/id_dsa ((nil)),
debug2: key: /home/backuper/.ssh/id_ecdsa ((nil)),
debug2: key: /home/backuper/.ssh/id_ed25519 ((nil)),
debug3: input_userauth_banner
------------------------------
*** WEBA Networks ***
------------------------------

debug1: Authentications that can continue: publickey,keyboard-interactive,password
debug3: start over, passed a different list publickey,keyboard-interactive,password
debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /home/backuper/.ssh/id_rsa
debug3: send_pubkey_test
debug2: we sent a publickey packet, wait for reply
debug1: Server accepts key: pkalg ssh-rsa blen 1047
debug2: input_userauth_pk_ok: fp 96??ad:52:aa:39:9f:0c:e3:c2:bb:f1:08:79:48:3d
debug3: sign_and_send_pubkey: RSA 96??ad:52:aa:39:9f:0c:e3:c2:bb:f1:08:79:48:3d
debug1: key_parse_private2: missing begin marker
debug1: read PEM private key done: type RSA
debug1: Authentication succeeded (publickey).
Authenticated to 10.44.0.15 ([10.44.0.15]:22).
debug2: fd 4 setting O_NONBLOCK
debug2: fd 5 setting O_NONBLOCK
debug2: fd 6 setting O_NONBLOCK
debug1: channel 0: new [client-session]
debug3: ssh_session2_open: channel_new: 0
debug2: channel 0: send open
debug1: Entering interactive session.
debug2: callback start
debug2: fd 3 setting TCP_NODELAY
debug3: packet_set_tos: set IP_TOS 0x08
debug2: client_session2_setup: id 0
debug1: Sending environment.
debug3: Ignored env XDG_SESSION_ID
debug3: Ignored env SHELL
debug3: Ignored env USER
debug3: Ignored env PATH
debug3: Ignored env PWD
debug1: Sending env LANG = ru_RU.UTF-8
debug2: channel 0: request env confirm 0
debug3: Ignored env HOME
debug3: Ignored env SHLVL
debug3: Ignored env LOGNAME
debug3: Ignored env XDG_RUNTIME_DIR
debug3: Ignored env _
debug3: Ignored env OLDPWD
debug1: Sending command: show configuration
debug2: channel 0: request exec confirm 1
debug2: callback done
debug2: channel 0: open confirm rwindow 16384 rmax 32768
debug2: channel 0: read<=0 rfd 4 len 0
debug2: channel 0: read failed
debug2: channel 0: close_read
debug2: channel 0: input open -> drain
debug2: channel 0: ibuf empty
debug2: channel 0: send eof
debug2: channel 0: input drain -> closed
debug2: channel_input_status_confirm: type 99 id 0
debug2: exec request accepted on channel 0
debug3: Received SSH2_MSG_IGNORE
debug2: channel 0: rcvd eof
debug2: channel 0: output open -> drain
debug2: channel 0: rcvd close
debug3: channel 0: will not send data after close
debug3: channel 0: will not send data after close
debug2: channel 0: obuf empty
debug2: channel 0: close_write
debug2: channel 0: output drain -> closed
debug2: channel 0: almost dead
debug2: channel 0: gc: notify user
debug2: channel 0: gc: user detached
debug2: channel 0: send close
debug2: channel 0: is dead
debug2: channel 0: garbage collecting
debug1: channel 0: free: client-session, nchannels 1
debug3: channel 0: status: The following connections are open:
#0 client-session (t4 r0 i3/0 o3/0 fd -1/-1 cc -1)

debug1: fd 0 clearing O_NONBLOCK
debug1: fd 1 clearing O_NONBLOCK
debug1: fd 2 clearing O_NONBLOCK
Transferred: sent 5712, received 2888 bytes, in 0.0 seconds
Bytes per second: sent 263664.4, received 133309.3
debug1: Exit status -1
----------------------------------

SSH Connect taking place , login-banner passed:
------------------------------
*** WEBA Networks ***
------------------------------
but no config-data received!

I use this servers for backup some other equipment with ssh-access (ex, Juniper EX/MX devices) with same schema (command "show config ..." over ssh-connect) and never has problem.

Problem appear only from backup config from Summit Ex460-48x.

I will hope for any idea for fix this issue.

RE: Problem with backup config over SSH

eyeV — Wed, 02 Sep 2015 10:14:00 GMT

Hi!
I'm going to reproduce your problem today. I've never thought about this method of backing up.

It's nice to see some russian names in Hub, namesake!

RE: Problem with backup config over SSH

Mikhail_Ivanov — Wed, 02 Sep 2015 12:46:00 GMT

This method is very simple - one string in script for one device, and very secure. Only one key-pair for each backup-server, no additional software, keys with 8192 bits works  Interesting feature - SSH-Client in Ex460 work correctly with manual shell-connect, but not work, when i execute command from script by cron. I create this script: -------------------- #!/bin/bash date='date "+%Y_%m_%d"'; # current date [YYYY_mm_dd] dir_open=/home/backuper/1/d/ # directory for store backups sftp_bin='/usr/bin/sftp -i /home/backuper/.ssh/id_rsa -r '; mkdir $dir_open/$date cd $dir_open/$date && mkdir Ex460Gate sync echo " ...BackUp All other servers ..."; ssh -vvv -o 'BatchMode yes' 10.44.0.15 "show configuration" > $dir_open/$date/Ex460Gate/ex460-48x_gate.conf 2>/home/backuper/1/1.log ssh -vvv -o 'BatchMode yes' 10.44.0.15 "show configuration detail" > $dir_open/$date/Ex460Gate/ex460-48x_gate-detail.conf 2>/home/backuper/1/2.log ---------------- When i run this script from shell manually - backup work correctly. When i run this script from crontab automaticaly - no backup taking place. Possibly some tweaks need - but i don't know. I use this script for many other servers and Juniper equipment and never see problem. Additionally, in this scheme backup-server less vulnerable in case of equipment or server's hack. P.S. About russian names - at first i try found help on russian IT-forums, but this issue similar more complex, and i post this message in official forum. I can make any logs and diagnostic tests, exclude reset config on Ex460 

RE: Problem with backup config over SSH

Drew_C — Wed, 02 Sep 2015 12:46:00 GMT

You're not the only one who wants a private message feature 🙂

RE: Problem with backup config over SSH

eyeV — Wed, 02 Sep 2015 12:46:00 GMT

It would be very useful!

RE: Problem with backup config over SSH

Mikhail_Ivanov — Wed, 02 Sep 2015 14:25:00 GMT

I have Jabber misha[at]weba.ru if need private conversation.
More prefer solving equipment issue in public.

Make config backups manually so exhaustive.

topic RE: Problem with backup config over SSH in ExtremeSwitching (EXOS/Switch Engine)

Problem with backup config over SSH

RE: Problem with backup config over SSH

RE: Problem with backup config over SSH

RE: Problem with backup config over SSH

RE: Problem with backup config over SSH

RE: Problem with backup config over SSH

RE: Problem with backup config over SSH

RE: Problem with backup config over SSH