SSH problems after updating to 30.7.1.11: kex_exchange_identification: read: Connection reset by peer Connection reset by...

[BRMS]

After updating a few x440-g2 test switches to version 30.7.1.11 im unable to ssh into the switch a day after booting the new firmware. the error-message i get is:

kex_exchange_identification: read: Connection reset by peer
Connection reset by 172.28.32.18 port 22

the only way to get ssh back is logging in via webfrontend, then using the cli to “restart process exsshd”. after that i can login again, im not sure how long however, since yesterday after updating the switches, ssh still worked.

the logs on the switches show the following:

08/31/2020 22:31:10.97 <Info:HAL.Card.Info> Switch is operational
08/31/2020 22:31:02.27 <Noti:exsshd.CfgMsgAuthCodeWeak> The configured message authentication code(s), hmac-sha1-96, hmac-md5, hmac-md5-96, hmac-sha1-96-etm@openssh.com, hmac-md5-etm@openssh.com, hmac-md5-96-etm@openssh.com, is/are weaker than what is recommended.
08/31/2020 22:31:02.27 <Noti:exsshd.CfgCipherWeak> The configured cipher(s), 3des-cbc, aes128-cbc, aes192-cbc, aes256-cbc, rijndael-cbc@lysator.liu.se, is/are weaker than what is recommended.
08/31/2020 22:31:02.27 <Noti:exsshd.CfgPubKeyAlgWeak> The configured public key algorithm(s), ssh-dss, is/are weaker than what is recommended.
08/31/2020 22:31:02.27 <Noti:exsshd.CfgKeyExchgAlgWeak> The configured key exchange algorithm(s), DH group 1, is/are weaker than what is recommended.
08/31/2020 22:31:02.08 <Info:SNMP.Master.InitDone> snmpMaster initialization complete
08/31/2020 22:31:02.04 <Erro:exsshd.LoadCfgMACUnsuprt> Load configuration of message authentication code hmac-ripemd160@openssh.com failed, since it is unsupported.
08/31/2020 22:31:02.03 <Erro:exsshd.LoadCfgMACUnsuprt> Load configuration of message authentication code hmac-ripemd160 failed, since it is unsupported.
08/31/2020 22:31:01.75 <Erro:exsshd.LoadCfgMACUnsuprt> Load configuration of message authentication code hmac-ripemd160-etm@openssh.com failed, since it is unsupported.
08/31/2020 22:31:01.70 <Warn:NetTools.SNTP.TxReqToSrvrFail> Failed to send SNTP request to server 10.231.130.5
08/31/2020 22:31:01.65 <Warn:NetTools.SNTP.TxReqToSrvrFail> Failed to send SNTP request to server 10.231.130.5
08/31/2020 22:31:01.63 <Erro:exsshd.LoadCfgCipherUnsuprt> Load configuration of cipher arcfour128 failed, since it is unsupported.
08/31/2020 22:31:01.62 <Erro:exsshd.LoadCfgCipherUnsuprt> Load configuration of cipher arcfour256 failed, since it is unsupported.
08/31/2020 22:31:01.52 <Erro:exsshd.LoadCfgCipherUnsuprt> Load configuration of cipher arcfour failed, since it is unsupported.
08/31/2020 22:31:01.32 <Warn:log.NVRAMFull> NVRAM is full, old messages are overwritten.
08/31/2020 22:31:01.20 <Erro:exsshd.LoadCfgCipherUnsuprt> Load configuration of cipher cast128-cbc failed, since it is unsupported.
08/31/2020 22:31:01.16 <Erro:exsshd.LoadCfgCipherUnsuprt> Load configuration of cipher blowfish-cbc failed, since it is unsupported.
08/31/2020 22:31:00.80 <Noti:exsshd.BndAccntKey> Bind user XXXXX to SSH public key XXXXX of SHA256 fingerprint f1:85:6f:fa:44:45:91:9a:0e:fb:9c:96:28:8b:fc:59:f9:ad:c6:3b:85:e4:62:b8:d5:65:2f:e9:2e:a2:f1:e4 successfully
08/31/2020 22:31:00.77 <Noti:exsshd.BndAccntKey> Bind user XXXXX to SSH public key XXXXX of SHA256 fingerprint f1:85:6f:fa:44:45:91:9a:0e:fb:9c:96:28:8b:fc:59:f9:ad:c6:3b:85:e4:62:b8:d5:65:2f:e9:2e:a2:f1:e4 successfully
08/31/2020 22:31:00.72 <Noti:exsshd.BndAccntKey> Bind user XXXXX to SSH public key XXXXX of SHA256 fingerprint 8e:6e:34:72:31:7d:5a:90:f3:65:cd:aa:b4:f3:f9:12:5c:8c:fd:72:1e:e8:35:98:b8:46:13:c5:4a:a4:71:8b successfully
08/31/2020 22:31:00.70 <Noti:exsshd.BndAccntKey> Bind user admin to SSH public key admin of SHA256 fingerprint 14:6b:7c:cd:c1:d5:f3:a8:18:dd:5a:25:6c:a9:3f:64:fa:5b:14:79:ff:1b:4a:5b:11:59:75:8b:4a:fb:17:75 successfully
08/31/2020 22:31:00.66 <Noti:exsshd.BndAccntKey> Bind user XXXXX to SSH public key XXXXX of SHA256 fingerprint 47:39:9f:43:c2:db:2b:1f:23:be:a0:12:84:3e:db:ab:0c:8f:31:8e:6f:23:e1:39:80:8c:ad:8b:78:44:0d:7c successfully
08/31/2020 22:31:00.33 <Info:AAA.LogSsh> Msg from Master : Existing host key with fingerprint SHA256:75:a0:98:21:ed:21:5f:63:12:8e:db:ff:45:f0:9f:1a:7d:b4:af:3d:c5:87:c5:f9:e6:bc:61:93:e5:1c:14:df loaded successfully
08/31/2020 22:31:00.25 <Info:AAA.LogSsh> Msg from Master : Generating RSA-2048 public key
08/31/2020 22:31:00.12 <Info:AAA.LogSsh> Msg from Master : Loaded Private Key of size 1679 from System
08/31/2020 22:30:50.90 <Noti:EPM.system_stable> System is stable. Change to warm reset mode
08/31/2020 22:30:50.25 <Erro:cm.sys.LoadApplCfgObjFail> "otm" application failed to load "otmGlobal" configuration object: No Action -- Insufficient License
                                                        
08/31/2020 22:30:46.20 <Noti:DM.Notice> Process ztpstack Deleted
08/31/2020 22:30:45.17 <Info:vlan.mac.gMacArrayDump> (hal) gMacTree[0]: 0:4:96:ae:79:4b refCnt:0 src:no peerId:0
08/31/2020 22:30:45.17 <Info:vlan.mac.gMacArrayDump> (hal) Dumping elements of gMacTree *******
08/31/2020 22:30:45.00 <Info:vlan.mac.gMacArrayDump> (esrp) gMacTree[0]: 0:4:96:ae:79:4b refCnt:0 src:no peerId:0
08/31/2020 22:30:45.00 <Info:vlan.mac.gMacArrayDump> (esrp) Dumping elements of gMacTree *******
08/31/2020 22:30:44.99 <Info:vlan.mac.gMacArrayDump> (netTools) gMacTree[0]: 0:4:96:ae:79:4b refCnt:0 src:no peerId:0
08/31/2020 22:30:44.99 <Info:vlan.mac.gMacArrayDump> (netTools) Dumping elements of gMacTree *******
08/31/2020 22:30:44.98 <Info:vlan.mac.gMacArrayDump> (mcmgr) gMacTree[0]: 0:4:96:ae:79:4b refCnt:0 src:no peerId:0
08/31/2020 22:30:44.98 <Info:vlan.mac.gMacArrayDump> (mcmgr) Dumping elements of gMacTree *******
08/31/2020 22:30:44.97 <Info:vlan.mac.gMacArrayDump> (vlan) gMacTree[0]: 0:4:96:ae:79:4b refCnt:0 src:no peerId:0
08/31/2020 22:30:44.97 <Info:vlan.mac.gMacArrayDump> (vlan) Dumping elements of gMacTree *******
08/31/2020 22:30:38.38 <Info:EPM.wdg_enable> Watchdog enabled
08/31/2020 22:30:33.68 <Noti:DM.Notice> Node State[3] = OPERATIONAL
08/31/2020 22:30:32.56 <Info:telnetd.info> **** telnetd started *****
08/31/2020 22:30:31.50 <Info:DOSProt.Init> DOS protect application started successfully
08/31/2020 22:30:31.21 <Info:tftpd.info> **** tftpd started *****
08/31/2020 22:30:31.17 <Info:HAL.Sys.Info> External Power Supply is disconnected.
08/31/2020 22:30:31.17 <Info:HAL.Sys.Info> Internal Power Supply is disconnected.
08/31/2020 22:30:31.11 <Noti:SNMP.Subagent.MstrRestrt> snmpMaster process has been restarted.
08/31/2020 22:30:31.05 <Info:SNMP.Subagent.InitDone> snmpSubagent initialization complete
08/31/2020 22:30:30.54 <Noti:DM.Notice> Node State[2] = STANDBY
08/31/2020 22:30:30.54 <Info:DM.Info> Node INIT DONE ....
08/31/2020 22:30:29.33 <Noti:DM.Notice> Node State[1] = INIT
08/31/2020 22:30:29.21 <Info:nl.init> Network Login framework has been initialized
08/31/2020 22:30:28.15 <Info:HAL.Sys.Info> Hal initialization done.
08/31/2020 22:30:27.28 <Info:HAL.Sys.Info> External Power Supply is powered off.
08/31/2020 22:30:27.28 <Info:HAL.Sys.Info> External Power Supply is present.
08/31/2020 22:30:27.28 <Info:HAL.Sys.Info> Internal Power Supply is powered off.
08/31/2020 22:30:27.28 <Info:HAL.Sys.Info> Internal Power Supply is present.
08/31/2020 22:30:27.28 <Noti:HAL.Sys.Notice> Module in fan slot 1 is inserted
08/31/2020 22:30:27.28 <Info:HAL.Card.Info> Bluetooth capability is not supported
08/31/2020 22:29:54.01 <Noti:log.serverStarted> The Event Management System logging server has started.
08/31/2020 22:29:53.85 <Info:HAL.Sys.Info> Starting hal initialization ....
08/31/2020 22:29:52.21 <Noti:DM.Notice> DM started
08/31/2020 22:29:50.84 <Noti:NM.StrtProc> The Node Manager (NM) has started processing.
08/31/2020 22:29:50.66 <Noti:EPM.start> EPM Started
08/31/2020 22:29:49.51 <Noti:EPM.wd_warm_reset> Changing to watchdog warm reset mode
08/31/2020 22:29:00.39 <Warn:EPM.all_shutdown> Shutting down all processes
08/31/2020 22:29:00.18 <Warn:EPM.reboot> User unknown: Rebooting with reason User requested switch reboot
08/31/2020 14:18:03.09 <Erro:AAA.RADIUS.serverNotInit> authentication server for management-access is not initialized
08/31/2020 14:16:19.65 <Warn:EPM.scheduled_reboot> User scheduled Next Reboot: Switch Mon Aug 31 22:29:00 2020

after restarting the process which solves the problem for now i get the following log entries:

09/01/2020 11:14:42.57 <Noti:exsshd.CfgMsgAuthCodeWeak> The configured message authentication code(s), hmac-sha1-96, hmac-md5, hmac-md5-96, hmac-sha1-96-etm@openssh.com, hmac-md5-etm@openssh.com, hmac-md5-96-etm@openssh.com, is/are weaker than what is recommended.
09/01/2020 11:14:42.57 <Noti:exsshd.CfgCipherWeak> The configured cipher(s), 3des-cbc, aes128-cbc, aes192-cbc, aes256-cbc, rijndael-cbc@lysator.liu.se, is/are weaker than what is recommended.
09/01/2020 11:14:42.57 <Noti:exsshd.CfgPubKeyAlgWeak> The configured public key algorithm(s), ssh-dss, is/are weaker than what is recommended.
09/01/2020 11:14:42.57 <Noti:exsshd.CfgKeyExchgAlgWeak> The configured key exchange algorithm(s), DH group 1, is/are weaker than what is recommended.
09/01/2020 11:14:41.88 <Noti:exsshd.BndAccntKey> Bind user XXXXX to SSH public key XXXXX of SHA256 fingerprint f1:85:6f:fa:44:45:91:9a:0e:fb:9c:96:28:8b:fc:59:f9:ad:c6:3b:85:e4:62:b8:d5:65:2f:e9:2e:a2:f1:e4 successfully
09/01/2020 11:14:41.87 <Noti:exsshd.BndAccntKey> Bind user XXXXX to SSH public key XXXXX of SHA256 fingerprint f1:85:6f:fa:44:45:91:9a:0e:fb:9c:96:28:8b:fc:59:f9:ad:c6:3b:85:e4:62:b8:d5:65:2f:e9:2e:a2:f1:e4 successfully
09/01/2020 11:14:41.87 <Noti:exsshd.BndAccntKey> Bind user XXXXX to SSH public key XXXXX of SHA256 fingerprint 8e:6e:34:72:31:7d:5a:90:f3:65:cd:aa:b4:f3:f9:12:5c:8c:fd:72:1e:e8:35:98:b8:46:13:c5:4a:a4:71:8b successfully
09/01/2020 11:14:41.87 <Noti:exsshd.BndAccntKey> Bind user XXXXX to SSH public key XXXXX of SHA256 fingerprint 14:6b:7c:cd:c1:d5:f3:a8:18:dd:5a:25:6c:a9:3f:64:fa:5b:14:79:ff:1b:4a:5b:11:59:75:8b:4a:fb:17:75 successfully
09/01/2020 11:14:41.87 <Noti:exsshd.BndAccntKey> Bind user XXXXX to SSH public key XXXXX of SHA256 fingerprint 47:39:9f:43:c2:db:2b:1f:23:be:a0:12:84:3e:db:ab:0c:8f:31:8e:6f:23:e1:39:80:8c:ad:8b:78:44:0d:7c successfully
09/01/2020 11:14:40.05 <Noti:EPM.Msg.proc_start> Requested process exsshd start
09/01/2020 11:14:39.99 <Noti:DM.Notice> Process exsshd Stopped
09/01/2020 11:14:39.71 <Noti:EPM.Msg.proc_shutdown> Requested process exsshd shutdown

[Stefan_K_]

Ah, now it’s there. 🙂

[Stefan_K_]

Am I blind? Where did you find it? It’s not in the Software Download Center, at least not for me.

[BRMS]

thanks. the patched version got released and i already installed it.

i found another bug btw.: this new login counter shown after logging in:

There have been 0 successful logins since last reboot and 0 failed logins since last successful login
No Prior Logins by this user since last reboot

does not account for logins which get authenticated via ssh-key. only if i login using password the counter gets incremented. i don't care for that functionality tbh, but maybe someone else does.

Have a nice day!

[BradP · Accepted solution]

Hello

 This will be fixed in the next patch, 30.7.1 Patch 1-23. The patch is expected to release any day now. 

Thank you,

 Brad