Okay this is a tricky one
In the past few months, all of my ubuntu 16.04 servers stopped accepting public key SSH authentication and will only accept passwords. Some of the things I tried:
- Auditing all file permissions related to ssh
- Recopying pubkeys to servers with ssh-copy
- Stopping sshd, apt-purge all SSH and associated packages, then rebooting and reinstalling-- which resets sshd_config as well as re-keys server
- Generating new keys on the client with large bits (4096)
- And almost every other thing I could find on the subject.
Couple of interesting points:
- All ubuntu servers were scheduled with unattended-upgrades to apply security updates. They all stopped working at around the same time. No other software changes were made
- All servers are virtualized across multiple platforms. Some on VMware guests, some in KVM unprivileged LXC containers, some as KVM guests.
- Brand new installs work fine.
- Debian based servers didn't have this failure.
- Debug logs don't give any permissions failures, kex failures, or any other clue, other than
ssh-connection method none [preauth]on non-working servers vs
ssh-connection method publickey [preauth]on working servers. (complete client and server logs below)
- Client machine, keys and software are exactly the same
- If I launch sshd via command line I can connect with public key just fine.
- Upgrading from 16.04 to 18.04 via do-release-upgrade doesn't solve the issue, it persists.
Ubuntu Client and server failure logs: https://pastebin.com/Tg3pu8Nw
Debian Client and server success (same client machine and keys): https://pastebin.com/aG5ddrQ5
Any help would be greatly appreciated.