ssh stops working on CentOS 8 (GCE VM)

Issues related to configuring your network
Post Reply
devlocalca
Posts: 10
Joined: 2019/06/28 05:58:41

ssh stops working on CentOS 8 (GCE VM)

Post by devlocalca » 2019/10/17 11:05:05

ssh stops working on CentOS 8 (GCE VM)

I had ssh working correctly with a pub/private key pair working just fine for two days on a GCE CentOS 8 VM, and it stopped working.

Now I get:

Code: Select all

Permission denied (publickey,gssapi-keyex,gssapi-with-mic).
So it was working, meaning everything was setup, configured correctly, with all permissions working correctly, and now it does not work, so it's not the configuration. Nothing changed in the configuration.

I have the same setup working on CentOS 7, and it just keeps going and going, has never gone down, stopped working or had a problem.

The configuration is correct, I checked file and directory permissions.

Code: Select all

$cd $HOME/.ssh
$ls -al .
Results:

Code: Select all

drwx------.  3 user group  156 Oct 17 10:11 .
drwxrwx---. 12 user group 4096 Oct 17 10:09 ..
-rw-------.  1 user group  197 Oct 17 10:09 authorized_keys
I generated a key like so (in cygwin), and put the .pub contents on the GCE VM in /home/user/.ssh/authorized_keys

Code: Select all

ssh-keygen -f id_ed25519 -o -a 100 -t ed25519 -C "from:cygwin"
It was working earlier in the day, and now it's not.

My connection script:

Code: Select all

#!/bin/bash
IP=x.x.x.x
LOCAL_KEYFILE="/c/home/user/.ssh/id_ed25519"
REMOTE_USER='user'
ssh -v -i $LOCAL_KEYFILE $REMOTE_USER@$IP
$systemctl status sshd.service

Code: Select all

● sshd.service - OpenSSH server daemon
   Loaded: loaded (/usr/lib/systemd/system/sshd.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2019-10-17 10:19:12 UTC; 17ms ago
     Docs: man:sshd(8)
           man:sshd_config(5)
 Main PID: 2504 (sshd)
    Tasks: 1 (limit: 26213)
   Memory: 1.1M
   CGroup: /system.slice/sshd.service
           └─2504 /usr/sbin/sshd -D -oCiphers=aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes256-ctr,aes256-cbc,aes128-gcm@openssh.com,aes128-ctr,aes128-cbc -oMACs=hmac-sha2-256-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-128-etm@openssh.c>
Oct 17 10:19:12 host_name systemd[1]: Starting OpenSSH server daemon...
Oct 17 10:19:12 host_name sshd[2504]: Server listening on 0.0.0.0 port 22.
Oct 17 10:19:12 host_name sshd[2504]: Server listening on :: port 22.
Oct 17 10:19:12 host_name systemd[1]: Started OpenSSH server daemon.
verbose output when trying to connect:

Code: Select all

OpenSSH_8.0p1, OpenSSL 1.1.1d  10 Sep 2019
debug1: Connecting to x.x.x.x [x.x.x.x] port 22.
debug1: Connection established.
debug1: identity file /g/home/user/.ssh/id_ed25519 type 3
debug1: identity file /g/home/user/.ssh/id_ed25519-cert type -1    <== I dunno what this is, I don't have this file local
debug1: Local version string SSH-2.0-OpenSSH_8.0
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.8
debug1: match: OpenSSH_7.8 pat OpenSSH* compat 0x04000000
debug1: Authenticating to 34.82.235.87:22 as 'proc'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256@libssh.org
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:nfWd1FFBjYKQ7G7j4hkftEfB8UowSKkwfJzC2T8UF1g
debug1: Host '34.82.235.87' is known and matches the ECDSA host key.
debug1: Found key in /home/<another-unrelated-user>/.ssh/known_hosts:11  <== why is it reference another user here?
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 134217728 blocks
debug1: Will attempt key: /g/home/user/.ssh/id_ed25519 ED25519 SHA256:7o0XJjVnZLuzi+fO1NUUkFKpwKqw87u208IBy53zHcI explicit
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,null>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug1: Next authentication method: publickey
debug1: Offering public key: /g/home/user/.ssh/id_ed25519 ED25519 SHA256:7o0XJjVnZLuzi+fO1NUUkFKpwKqw87u208IBy53zHcI explicit
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug1: No more authentication methods to try.
user@x.x.x.x: Permission denied (publickey,gssapi-keyex,gssapi-with-mic).
---
I found some articles that said I should run these commands, so I did:

Code: Select all

restorecon -Rv /home/user/.ssh/
restorecon -Rv /home/user/.ssh/authorized_keys
semanage fcontext -a -t ssh_home_t "/home/user/.ssh/authorized_keys"
and then after running each one, I restarted sshd, and tried to connect, and got the same results (same error message).
restart command:

Code: Select all

systemctl restart sshd.service
Any ideas would be greatly appreciated, this is baffling.

I turned off selinux with:

Code: Select all

#setenforce 0
and in /etc/selinux/config, I set SELINUX=disabled
and restarted the VM, and no such luck.

Nothing has changed in the GCE firewall settings, port 22 is open, and I can connect to a VM using configured with the same network.

I can connect via ssh, when open a window in the browser through:

https://console.cloud.google.com/comput ... oject-name>

---

hunter86_bg
Posts: 2019
Joined: 2015/02/17 15:14:33
Location: Bulgaria
Contact:

Re: ssh stops working on CentOS 8 (GCE VM)

Post by hunter86_bg » 2019/10/17 15:08:43

Have you tried to run the ssh daemon with the same options and increased verbosity /debug level ?

Post Reply