2

TL;DR
I rsynced a few directories from /home/ubuntu (set to 777) of one remote server to my new AWS EC2 instance. I am now locked out of sshing into it by a Permission denied (publickey) error.

I am in the process of migrating my production environment from SoftLayer to AWS.

I had to rsync several directories to EC2 (EBS) and in the process, I did transfer a few directories from the old /home/ubuntu/ to my current EC2 instance's /home/ubuntu/.

My rsync command (on destination) looked like this.

ubuntu@[aws.remote.ec2]:~$ sudo rsync --include 'dir1' --include '*.sh' --include '.py' --include 'api_logs' --include 'database_backups' --exclude '*' -avz -e "ssh -p $portNumber" ubuntu@[softlayer.remote]:/home/ubuntu/ /home/ubuntu/

Files were successfully transferred. When I tried to ssh into my EC2 the next time I got a Permission denied (publickey) with the following log with the ssh -v option: (I have masked private info like IP with {} in the log below)

OpenSSH_7.2p2 Ubuntu-4ubuntu2.2, OpenSSL 1.0.2g  1 Mar 2016
debug1: Reading configuration data /home/{localuser}/.ssh/config
debug1: /home/{localuser}/.ssh/config line 1: Applying options for aws-fr-
ec2
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug1: Connecting to {aws.ec2.ip} [{aws.ec2.ip}] port 22.
debug1: Connection established.
debug1: key_load_public: No such file or directory
debug1: identity file /home/{localuser}/Documents/AWS-Files/EC2-FR.pem type 
-1
debug1: key_load_public: No such file or directory
debug1: identity file /home/{localuser}/Documents/AWS-Files/EC2-FR.pem-cert 
type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.2
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.2p2 
Ubuntu-4ubuntu2.2
debug1: match: OpenSSH_7.2p2 Ubuntu-4ubuntu2.2 pat OpenSSH* compat 
0x04000000
debug1: Authenticating to {aws.ec2.ip}:22 as 'ubuntu'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: [email protected]
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: [email protected] MAC: 
<implicit> compression: none
debug1: kex: client->server cipher: [email protected] MAC: 
<implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 
SHA256:g3nWVGmjJYVrNrwsDJMhzbLSw0FzBOLoUx80seD9qIs
debug1: Host '{aws.ec2.ip}' is known and matches the ECDSA host key.
debug1: Found key in /home/{localhost}/.ssh/known_hosts:11
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Trying private key: /home/{localhost}/Documents/AWS-Files/EC2-
FR.pem
debug1: Authentications that can continue: publickey
debug1: No more authentication methods to try.
Permission denied (publickey).

I did stumble upon this question but without help. I also found this thread on the AWS Forum.

I did follow the steps listed:

Posted by: mary@AWS:
Could you please verify the permissions on the /home/ubuntu/.ssh directory and files contained in it on this instance?

To verify the permissions, you can stop the instance and detach the root volume (make note of the device that it's attached to). Then attach the volume to another instance on an available device. Create a mount point, such as /fixroot, if needed and mount the device to this mount point. Once mounted, cd to /fixroot/home/ec2-user and check the directory and file permissions. The .ssh directory should allow rwx for the user (owner) and the files should be readable only by the user.

Another thing to check while you are there is that the known_hosts file doesn't have duplicate entries for the client you are attempting to connect from.

Once you have done this, you can unmount the volume and detach it from the instance. Then attach it back to the original instance to the device that you noted in the first step and start the instance.

As well as

Posted by: yromanenko:
turns out it was the relaxed permissions on the home/ubuntu folder rather than ssh. I was able to fix it by detaching the root volume and fixing permissions. The following video was very helpful in guiding me through the steps:

http://d2930476l2fsmh.cloudfront.net/LostKeypairRecoveryOfLinuxInstance.mp4

I spawned a new t2.micro instance and followed Mary's steps to confirm the permissions and yromanenko's resolution to set 755 on the /home/ubuntu directory.

I reattached the problematic EBS device back to the first EC2 as /dev/sda1 and tried only to fail with the same Permission denied (publickey) error!!

Consequently, now I am getting the same error on the secondary t2.micro instance. :(

Any help would be appreciated!

2
  • It has to be "turtles all the way down" - you can check all the intervening ownerships and permissions with namei -l /home/ubuntu/.ssh/authorized_keys for example Nov 8, 2017 at 16:29
  • unfortunately, I've been hit with the same issue today. Please add an answer if you've successfully solved it.
    – kmonsoor
    Jan 22, 2018 at 21:05

1 Answer 1

0

I had this exact issue. I was using an Amazon EC2 instance so I could spin up another and identify the differences. The broken machine had /home/user permissions 777, the not broken machine had 700. For some inexplicable reason, rsyncing to /home/user changes the permissions of /home/user to 777. Apparently SSH requires /home/user to be 700 for security reasons. This also explains why its turtles all the way down, rsync again and it's broken again.

If you are fortunate enough to have access to the directory in some other way,

chmod 700 /home/user

fixes it.

In the future, rsync to a subdirectory of /home/user.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .