TL;DR
Irsync
ed a few directories from/home/ubuntu
(set to777
) of one remote server to my new AWS EC2 instance. I am now locked out ofssh
ing into it by aPermission denied (publickey)
error.
I am in the process of migrating my production environment from SoftLayer to AWS.
I had to rsync
several directories to EC2 (EBS) and in the process, I did transfer a few directories from the old /home/ubuntu/
to my current EC2 instance's /home/ubuntu/
.
My rsync
command (on destination) looked like this.
ubuntu@[aws.remote.ec2]:~$ sudo rsync --include 'dir1' --include '*.sh' --include '.py' --include 'api_logs' --include 'database_backups' --exclude '*' -avz -e "ssh -p $portNumber" ubuntu@[softlayer.remote]:/home/ubuntu/ /home/ubuntu/
Files were successfully transferred. When I tried to ssh
into my EC2 the next time I got a Permission denied (publickey)
with the following log with the ssh -v
option: (I have masked private info like IP with {} in the log below)
OpenSSH_7.2p2 Ubuntu-4ubuntu2.2, OpenSSL 1.0.2g 1 Mar 2016
debug1: Reading configuration data /home/{localuser}/.ssh/config
debug1: /home/{localuser}/.ssh/config line 1: Applying options for aws-fr-
ec2
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug1: Connecting to {aws.ec2.ip} [{aws.ec2.ip}] port 22.
debug1: Connection established.
debug1: key_load_public: No such file or directory
debug1: identity file /home/{localuser}/Documents/AWS-Files/EC2-FR.pem type
-1
debug1: key_load_public: No such file or directory
debug1: identity file /home/{localuser}/Documents/AWS-Files/EC2-FR.pem-cert
type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.2
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.2p2
Ubuntu-4ubuntu2.2
debug1: match: OpenSSH_7.2p2 Ubuntu-4ubuntu2.2 pat OpenSSH* compat
0x04000000
debug1: Authenticating to {aws.ec2.ip}:22 as 'ubuntu'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: [email protected]
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: [email protected] MAC:
<implicit> compression: none
debug1: kex: client->server cipher: [email protected] MAC:
<implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256
SHA256:g3nWVGmjJYVrNrwsDJMhzbLSw0FzBOLoUx80seD9qIs
debug1: Host '{aws.ec2.ip}' is known and matches the ECDSA host key.
debug1: Found key in /home/{localhost}/.ssh/known_hosts:11
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Trying private key: /home/{localhost}/Documents/AWS-Files/EC2-
FR.pem
debug1: Authentications that can continue: publickey
debug1: No more authentication methods to try.
Permission denied (publickey).
I did stumble upon this question but without help. I also found this thread on the AWS Forum.
I did follow the steps listed:
Posted by: mary@AWS:
Could you please verify the permissions on the /home/ubuntu/.ssh directory and files contained in it on this instance?
To verify the permissions, you can stop the instance and detach the root volume (make note of the device that it's attached to). Then attach the volume to another instance on an available device. Create a mount point, such as /fixroot, if needed and mount the device to this mount point. Once mounted, cd to /fixroot/home/ec2-user and check the directory and file permissions. The .ssh directory should allow rwx for the user (owner) and the files should be readable only by the user.
Another thing to check while you are there is that the known_hosts file doesn't have duplicate entries for the client you are attempting to connect from.
Once you have done this, you can unmount the volume and detach it from the instance. Then attach it back to the original instance to the device that you noted in the first step and start the instance.
As well as
Posted by: yromanenko:
turns out it was the relaxed permissions on the home/ubuntu folder rather than ssh. I was able to fix it by detaching the root volume and fixing permissions. The following video was very helpful in guiding me through the steps:
http://d2930476l2fsmh.cloudfront.net/LostKeypairRecoveryOfLinuxInstance.mp4
I spawned a new t2.micro
instance and followed Mary
's steps to confirm the permissions and yromanenko
's resolution to set 755
on the /home/ubuntu
directory.
I reattached the problematic EBS device back to the first EC2 as /dev/sda1
and tried only to fail with the same Permission denied (publickey)
error!!
Consequently, now I am getting the same error on the secondary t2.micro
instance. :(
Any help would be appreciated!
namei -l /home/ubuntu/.ssh/authorized_keys
for example