SSH -vvv password hang

[charlesgmoore]

Weird one here, we have about 30 of our existing CentOS boxes that are no longer allowing us to log in using password authentication.
We do know that if we uninstall/reinstall ssh it will work, the odd thing is that the port is listening, if we generate the RSA and use the RSA authentication method it will work.

output of ssh -vvv

OpenSSH_5.8p1 Debian-1ubuntu3, OpenSSL 0.9.8o 01 Jun 2010
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to stimulus.whittrio.com [65.107.59.66] port 22.
debug1: Connection established.
debug1: identity file /home/charles/.ssh/id_rsa type -1
debug1: identity file /home/charles/.ssh/id_rsa-cert type -1
debug1: identity file /home/charles/.ssh/id_dsa type -1
debug1: identity file /home/charles/.ssh/id_dsa-cert type -1
debug1: identity file /home/charles/.ssh/id_ecdsa type -1
debug1: identity file /home/charles/.ssh/id_ecdsa-cert type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_3.6.1p2
debug1: match: OpenSSH_3.6.1p2 pat OpenSSH_3.*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.8p1 Debian-1ubuntu3
debug2: fd 3 setting O_NONBLOCK
debug3: load_hostkeys: loading entries for host "stimulus.whittrio.com" from file "/home/charles/.ssh/known_hosts"
debug3: load_hostkeys: found key type RSA in file /home/charles/.ssh/known_hosts:201
debug3: load_hostkeys: loaded 1 keys
debug3: order_hostkeyalgs: prefer hostkeyalgs: ssh-rsa-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-rsa
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit: ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-rsa,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-dss-cert-v00@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: mac_setup: found hmac-md5
debug1: kex: server->client aes128-ctr hmac-md5 none
debug2: mac_setup: found hmac-md5
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug2: dh_gen_key: priv key bits set: 132/256
debug2: bits set: 511/1024
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Server host key: RSA 36:2f:1f:87:55:a3:a5:10:1a:d0:da:d9:24:40:52:18
debug3: load_hostkeys: loading entries for host "stimulus.whittrio.com" from file "/home/charles/.ssh/known_hosts"
debug3: load_hostkeys: found key type RSA in file /home/charles/.ssh/known_hosts:201
debug3: load_hostkeys: loaded 1 keys
debug3: load_hostkeys: loading entries for host "65.107.59.66" from file "/home/charles/.ssh/known_hosts"
debug3: load_hostkeys: found key type RSA in file /home/charles/.ssh/known_hosts:202
debug3: load_hostkeys: loaded 1 keys
debug1: Host 'stimulus.whittrio.com' is known and matches the RSA host key.
debug1: Found key in /home/charles/.ssh/known_hosts:201
debug2: bits set: 505/1024
debug1: ssh_rsa_verify: signature correct
debug2: kex_derive_keys
debug2: set_newkeys: mode 1
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug2: set_newkeys: mode 0
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug2: key: /home/charles/.ssh/id_rsa ((nil))
debug2: key: /home/charles/.ssh/id_dsa ((nil))
debug2: key: /home/charles/.ssh/id_ecdsa ((nil))
debug1: Authentications that can continue: publickey,password,keyboard-interactive
debug3: start over, passed a different list publickey,password,keyboard-interactive
debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Trying private key: /home/charles/.ssh/id_rsa
debug3: no such identity: /home/charles/.ssh/id_rsa
debug1: Trying private key: /home/charles/.ssh/id_dsa
debug3: no such identity: /home/charles/.ssh/id_dsa
debug1: Trying private key: /home/charles/.ssh/id_ecdsa
debug3: no such identity: /home/charles/.ssh/id_ecdsa
debug2: we did not send a packet, disable method
debug3: authmethod_lookup keyboard-interactive
debug3: remaining preferred: password
debug3: authmethod_is_enabled keyboard-interactive
debug1: Next authentication method: keyboard-interactive
debug2: userauth_kbdint
debug2: we sent a keyboard-interactive packet, wait for reply
debug1: Authentications that can continue: publickey,password,keyboard-interactive
debug3: userauth_kbdint: disable: no info_req_seen
debug2: we did not send a packet, disable method
debug3: authmethod_lookup password
debug3: remaining preferred:
debug3: authmethod_is_enabled password
debug1: Next authentication method: password
root@stimulus.whittrio.com's password:
debug3: packet_send2: adding 48 (len 62 padlen 18 extra_pad 64)
debug2: we sent a password packet, wait for reply

[pschaff]

Do you have "PasswordAuthentication yes" in /etc/ssh/sshd_config ? Does it work if SELinux is put in permissive mode with "setenforce 0"? Does it work if you are coming from a CentOS system rather than Debian?

[chagen]

I am seeing this on my Centos 5 system as well. I also have a friend who saw this exact behavior on an Ubuntu 10 system.

I noticed our system trying to access the address 659866826.9w7vcuctes.info via dns while I was trying to ssh to the box using a user/passwd combo setup.

On a couple systems ssh started working again and I do not see any odd dns requests going through when trying to ssh in.

I did notice that there was not an update for SSHD available, however, I removed it and re-installed and the md5sum on the new file is not the same as the old. It is the same on my non affected servers.

Chris.

[pschaff]

Welcome to the CentOS fora. Please see the recommended reading for new users linked in my signature.

The difference may be due to prelinking. The definitive test is [code]rpm -Va openssh\*[/code]
If you have questions please start a new Topic for your issue to get the attention you need, providing a link to this one if required for context, rather than hijacking this one.

[charlesgmoore]

We beat our heads against the wall for awhile and just decided to reinstall. We also noticed the MD5 issue but could find no solution.

Reinstall I guess

[pschaff]

There is no solution to a non-problem. See [url=https://www.google.com/search?q=rpm+prelink+site%3Aredhat.com&ie=utf-8&oe=utf-8&aq=t&rls=org.mozilla:en-US:unofficial&client=firefox-a]rpm prelink site:redhat.com[/url]. Reinstalling is much to RedmondOS-like a solution for my taste, unless you have been hacked or [i][b]really[/b][/i] messed up.