Okay this is a tricky one

In the past few months, all of my ubuntu 16.04 servers stopped accepting public key SSH authentication and will only accept passwords. Some of the things I tried:

  1. Auditing all file permissions related to ssh
  2. Recopying pubkeys to servers with ssh-copy
  3. Stopping sshd, apt-purge all SSH and associated packages, then rebooting and reinstalling-- which resets sshd_config as well as re-keys server
  4. Generating new keys on the client with large bits (4096)
  5. And almost every other thing I could find on the subject.

Couple of interesting points:

  1. All ubuntu servers were scheduled with unattended-upgrades to apply security updates. They all stopped working at around the same time. No other software changes were made
  2. All servers are virtualized across multiple platforms. Some on VMware guests, some in KVM unprivileged LXC containers, some as KVM guests.
  3. Brand new installs work fine.
  4. Debian based servers didn't have this failure.
  5. Debug logs don't give any permissions failures, kex failures, or any other clue, other than ssh-connection method none [preauth] on non-working servers vs ssh-connection method publickey [preauth] on working servers. (complete client and server logs below)
  6. Client machine, keys and software are exactly the same
  7. If I launch sshd via command line I can connect with public key just fine.
  8. Upgrading from 16.04 to 18.04 via do-release-upgrade doesn't solve the issue, it persists.

Ubuntu Client and server failure logs: https://pastebin.com/Tg3pu8Nw

Debian Client and server success (same client machine and keys): https://pastebin.com/aG5ddrQ5

Any help would be greatly appreciated.


Welp, figured I'd let everyone know just in case they ran into the same issue. Seems that the firewall appliances that we use were intercepting and deep-scanning SSH. That was breaking public key auth. Disabled the SSH deep inspection and the issue was resolved.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.