Problem with backup config over SSH

  • 0
  • 1
  • Problem
  • Updated 3 years ago
  • Not a Problem
Problem appear when make automated config-backups over SSH-protocol with key-based authentication.
Photo of Mikhail Ivanov

Mikhail Ivanov

  • 102 Points 100 badge 2x thumb

Posted 3 years ago

  • 0
  • 1
Photo of Mikhail Ivanov

Mikhail Ivanov

  • 102 Points 100 badge 2x thumb
How to reproduce:
I have Extreme Summit x460-48x switch with XOS  15.3.3.5 (patch1-4) with ssh-module.
I configured user `backuper` for key-based auth in ssh:

----------------------------------
create account admin backuper encrypted "<HASH>"
create sshd2 user-key backuper_ssh_pubkey AAAA<KEY-1-PUB-BODY>
create sshd2 user-key backuper2_ssh_pubkey AAAA<KEY-2-PUB-BODY>
configure sshd2 user-key backuper_ssh_pubkey add user backuper
configure sshd2 user-key backuper2_ssh_pubkey add user backuper
----------------------------------



User `backuper` contain two public-keys - each key for each backup-server.
I make backups from linux-server (Fedora 21) with this command:

----------------------------------
ssh 10.44.0.15 "show configuration" > ex460-48x_1.conf 2>/dev/null
----------------------------------


And have strange behavior - when i running this command in bash/sh shell manually, command work correctly, and configuration-dump stored in ex460-48x_1.conf .

But when i running this command from bash-script, executed by crontab every night, the result file contain only 2 bytes 0x0D 0x0A.

I turn on debug-log for ssh-client in my backup-script:

----------------------------------
ssh -v -v -v 10.44.0.15 "show configuration" 1> ex460-48x_gate.conf 2> ex460-48x_gate.log
----------------------------------



Detailed SSH-log, writed by cron:


----------------------------------
OpenSSH_6.6.1, OpenSSL 1.0.1k-fips 8 Jan 2015
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 51: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to 10.44.0.15.225 [10.44.0.15] port 22.
debug1: Connection established.
debug3: Incorrect RSA1 identifier
debug3: Could not load "/home/backuper/.ssh/id_rsa" as a RSA1 public key
debug1: identity file /home/backuper/.ssh/id_rsa type 1
debug1: identity file /home/backuper/.ssh/id_rsa-cert type -1
debug1: identity file /home/backuper/.ssh/id_dsa type -1
debug1: identity file /home/backuper/.ssh/id_dsa-cert type -1
debug1: identity file /home/backuper/.ssh/id_ecdsa type -1
debug1: identity file /home/backuper/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/backuper/.ssh/id_ed25519 type -1
debug1: identity file /home/backuper/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.6.1
debug1: Remote protocol version 2.0, remote software version 4.1.2 SSH Secure Shell Toolkit
debug1: no match: 4.1.2 SSH Secure Shell Toolkit
debug2: fd 3 setting O_NONBLOCK
debug3: load_hostkeys: loading entries for host "10.44.0.15" from file "/home/backuper/.ssh/known_hosts"
debug3: load_hostkeys: found key type DSA in file /home/backuper/.ssh/known_hosts:7
debug3: load_hostkeys: loaded 1 keys
debug3: order_hostkeyalgs: prefer hostkeyalgs: ssh-dss-cert-v01@openssh.com,ssh-dss-cert-v00@openssh.com,ssh-dss
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-dss-cert-v01@openssh.com,ssh-dss-cert-v00@openssh.com,ssh-dss,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,ssh-rsa
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: hmac-md5-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-sha1,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-sha1,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: kex_parse_kexinit: diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-dss
debug2: kex_parse_kexinit: aes256-cbc,aes192-cbc,aes128-cbc,twofish256-cbc,twofish-cbc,twofish192-cbc,twofish128-cbc,blowfish-cbc,3des-cbc
debug2: kex_parse_kexinit: aes256-cbc,aes192-cbc,aes128-cbc,twofish256-cbc,twofish-cbc,twofish192-cbc,twofish128-cbc,blowfish-cbc,3des-cbc
debug2: kex_parse_kexinit: hmac-sha1,hmac-md5,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-sha1,hmac-md5,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none
debug2: kex_parse_kexinit: none
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: mac_setup: setup hmac-md5
debug1: kex: server->client aes128-cbc hmac-md5 none
debug2: mac_setup: setup hmac-md5
debug1: kex: client->server aes128-cbc hmac-md5 none
debug1: kex: diffie-hellman-group1-sha1 need=16 dh_need=16
debug1: kex: diffie-hellman-group1-sha1 need=16 dh_need=16
debug2: bits set: 526/1024
debug1: sending SSH2_MSG_KEXDH_INIT
debug1: expecting SSH2_MSG_KEXDH_REPLY
debug1: Server host key: DSA 9a:a5:eb:a2:4b:22:d8:15:ab:f0:c7:da:a3:f1:3a:56
debug3: load_hostkeys: loading entries for host "10.44.0.15" from file "/home/backuper/.ssh/known_hosts"
debug3: load_hostkeys: found key type DSA in file /home/backuper/.ssh/known_hosts:7
debug3: load_hostkeys: loaded 1 keys
debug1: Host '10.44.0.15' is known and matches the DSA host key.
debug1: Found key in /home/backuper/.ssh/known_hosts:7
debug2: bits set: 524/1024
debug1: ssh_dss_verify: signature correct
debug2: kex_derive_keys
debug2: set_newkeys: mode 1
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug2: set_newkeys: mode 0
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug2: key: /home/backuper/.ssh/id_rsa (0x55c49f0af290),
debug2: key: /home/backuper/.ssh/id_dsa ((nil)),
debug2: key: /home/backuper/.ssh/id_ecdsa ((nil)),
debug2: key: /home/backuper/.ssh/id_ed25519 ((nil)),
debug3: input_userauth_banner
------------------------------
   *** WEBA Networks ***
------------------------------

debug1: Authentications that can continue: publickey,keyboard-interactive,password
debug3: start over, passed a different list publickey,keyboard-interactive,password
debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /home/backuper/.ssh/id_rsa
debug3: send_pubkey_test
debug2: we sent a publickey packet, wait for reply
debug1: Server accepts key: pkalg ssh-rsa blen 1047
debug2: input_userauth_pk_ok: fp 96:ee:ad:52:aa:39:9f:0c:e3:c2:bb:f1:08:79:48:3d
debug3: sign_and_send_pubkey: RSA 96:ee:ad:52:aa:39:9f:0c:e3:c2:bb:f1:08:79:48:3d
debug1: key_parse_private2: missing begin marker
debug1: read PEM private key done: type RSA
debug1: Authentication succeeded (publickey).
Authenticated to 10.44.0.15 ([10.44.0.15]:22).
debug2: fd 4 setting O_NONBLOCK
debug2: fd 5 setting O_NONBLOCK
debug2: fd 6 setting O_NONBLOCK
debug1: channel 0: new [client-session]
debug3: ssh_session2_open: channel_new: 0
debug2: channel 0: send open
debug1: Entering interactive session.
debug2: callback start
debug2: fd 3 setting TCP_NODELAY
debug3: packet_set_tos: set IP_TOS 0x08
debug2: client_session2_setup: id 0
debug1: Sending environment.
debug3: Ignored env XDG_SESSION_ID
debug3: Ignored env SHELL
debug3: Ignored env USER
debug3: Ignored env PATH
debug3: Ignored env PWD
debug1: Sending env LANG = ru_RU.UTF-8
debug2: channel 0: request env confirm 0
debug3: Ignored env HOME
debug3: Ignored env SHLVL
debug3: Ignored env LOGNAME
debug3: Ignored env XDG_RUNTIME_DIR
debug3: Ignored env _
debug3: Ignored env OLDPWD
debug1: Sending command: show configuration
debug2: channel 0: request exec confirm 1
debug2: callback done
debug2: channel 0: open confirm rwindow 16384 rmax 32768
debug2: channel 0: read<=0 rfd 4 len 0
debug2: channel 0: read failed
debug2: channel 0: close_read
debug2: channel 0: input open -> drain
debug2: channel 0: ibuf empty
debug2: channel 0: send eof
debug2: channel 0: input drain -> closed
debug2: channel_input_status_confirm: type 99 id 0
debug2: exec request accepted on channel 0
debug3: Received SSH2_MSG_IGNORE
debug2: channel 0: rcvd eof
debug2: channel 0: output open -> drain
debug2: channel 0: rcvd close
debug3: channel 0: will not send data after close
debug3: channel 0: will not send data after close
debug2: channel 0: obuf empty
debug2: channel 0: close_write
debug2: channel 0: output drain -> closed
debug2: channel 0: almost dead
debug2: channel 0: gc: notify user
debug2: channel 0: gc: user detached
debug2: channel 0: send close
debug2: channel 0: is dead
debug2: channel 0: garbage collecting
debug1: channel 0: free: client-session, nchannels 1
debug3: channel 0: status: The following connections are open:
  #0 client-session (t4 r0 i3/0 o3/0 fd -1/-1 cc -1)

debug1: fd 0 clearing O_NONBLOCK
debug1: fd 1 clearing O_NONBLOCK
debug1: fd 2 clearing O_NONBLOCK
Transferred: sent 5712, received 2888 bytes, in 0.0 seconds
Bytes per second: sent 263664.4, received 133309.3
debug1: Exit status -1
----------------------------------





SSH Connect taking place , login-banner passed:
------------------------------
   *** WEBA Networks ***
------------------------------
but no config-data received!


I use this servers for backup some other equipment with ssh-access (ex, Juniper EX/MX devices) with same schema (command "show config ..." over ssh-connect) and never has problem.

Problem appear only from backup config from Summit Ex460-48x.

I will hope for any idea for fix this issue.
(Edited)
Photo of eyeV

eyeV

  • 2,484 Points 2k badge 2x thumb
Hi!
I'm going to reproduce your problem today. I've never thought about this method of backing up.

It's nice to see some russian names in Hub, namesake!
Photo of Mikhail Ivanov

Mikhail Ivanov

  • 102 Points 100 badge 2x thumb
This method is very simple - one string in script for one device, and very secure.
Only one key-pair for each backup-server, no additional software, keys with 8192 bits works =)

Interesting feature - SSH-Client in Ex460 work correctly with manual shell-connect, but not work, when i execute command from script by cron.

I create this script:

--------------------
#!/bin/bash

date=`date "+%Y_%m_%d"`; # current date [YYYY_mm_dd]
dir_open=/home/backuper/1/d/ # directory for store backups
sftp_bin='/usr/bin/sftp -i /home/backuper/.ssh/id_rsa -r ';

mkdir $dir_open/$date
cd $dir_open/$date && mkdir Ex460Gate
sync

echo " ...BackUp All other servers ...";

ssh -vvv -o 'BatchMode yes' 10.44.0.15 "show configuration" > $dir_open/$date/Ex460Gate/ex460-48x_gate.conf 2>/home/backuper/1/1.log
ssh -vvv -o 'BatchMode yes' 10.44.0.15 "show configuration detail" > $dir_open/$date/Ex460Gate/ex460-48x_gate-detail.conf 2>/home/backuper/1/2.log
----------------

When i run this script from shell manually - backup work correctly.
When i run this script from crontab automaticaly - no backup taking place.

Possibly some tweaks need - but i don't know. I use this script for many other servers and Juniper equipment and never see problem.

Additionally, in this scheme backup-server less vulnerable in case of equipment or server's hack.

P.S. About russian names - at first i try found help on russian IT-forums, but this issue similar more complex, and i post this message in official forum.

I can make any logs and diagnostic tests, exclude reset config on Ex460 =)
Photo of eyeV

eyeV

  • 2,484 Points 2k badge 2x thumb
Offtop: It is a pity that the forum has no private messages option. It would be interesting to chat with colleague from Russia.
Photo of Drew C.

Drew C., Community Manager

  • 37,366 Points 20k badge 2x thumb
You're not the only one who wants a private message feature :)
Photo of eyeV

eyeV

  • 2,484 Points 2k badge 2x thumb
It would be very useful!
Photo of Mikhail Ivanov

Mikhail Ivanov

  • 102 Points 100 badge 2x thumb
I have Jabber misha[at]weba.ru if need private conversation.
More prefer solving equipment issue in public.

Make config backups manually so exhaustive.