24

I make frequent backups to a local drive which I want to sync daily to a remote server.

The target server is configured for SSH key (no password) access only. Since my primary SSH key for that server is passphrase-protected, I've created a second SSH key (not passphrase protected) + user to use for unattended backups - this way I do not have to be present to enter my passphrase when cron runs.

I'm using cron and rsync, and all of the commands work individually, but fail when combined.

The furthest I've got while troubleshooting is running

env -i sh -c "rsync -lrstRO --delete --exclude 'lost+found' /Backups/auto-daily-backups/./ [email protected]:/backups/desktop/"

which returns the error

Permission denied (publickey).
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: unexplained error (code 255) at io.c(226) [sender=3.1.0]

Any tips on how to troubleshoot this further?


Here's what I've tried so far and I'm out of ideas:

  1. Cron is definitely running ps aux | grep cron
  2. Nothing unusual in /var/log/syslog Sep 7 13:22:01 desktop CRON[6735]: (tom) CMD (sh /home/tom/Documents/Scripts/offsite-backup)

  3. SSH in Terminal to remote server as the backup user works ssh [email protected]

  4. Running the command in Terminal works perfectly rsync -lrstRO --delete --exclude 'lost+found' /Backups/auto-daily-backups/./ [email protected]:/backups/desktop/
  5. Manually specifying the path to the backups-user key has no effect rsync -lrstRO --delete --exclude 'lost+found' -e 'ssh -i /home/tom/.ssh/backups-only' /Backups/auto-daily-backups/./ [email protected]:/backups/desktop/

  6. Replacing the non-functioning command with a simple test command works echo "Hello world" > ~/Desktop/test.txt

  7. Shouting/swearing at the computer had no effect (but made me feel better temporarily).


Edit 1:

Here's my crontab file and the script it calls.

...
# m h  dom mon dow   command
MAILTO=""
* * * * * sh /home/tom/Documents/Scripts/offsite-backup

and

#!/bin/bash

rsync -lrstRO --delete --exclude 'lost+found' /Backups/auto-daily-backups/./ [email protected]:/backups/desktop/

Edit 2:

Just to clarify, /var/log/auth.log on the target server contains the line Sep 11 08:23:01 <hostname> CRON[24421]: pam_unix(cron:session): session closed for user root This is confusing because I'm no longer running cron every minute locally, but a new entry still appears every minute in the server logs. Crontab files for all users (including root) on the server are empty & do nothing.

Also, user 'backups-only' was created only on the server and with limited rights, with a dedicated SSH key copied to my desktop machine. I'm assuming this is the way to go because everything works when running the commands manually.

The crontab file posted above is for me, user 'tom' on my desktop machine. My intent is to have it call the script which should log in to the server as user 'backups-only'. I just tried running the backup script (rather than the command inside it) and it successfully connected & worked. I ran it on my desktop as user 'tom', same user who created the cron job that won't work. Here's the output from the server log corresponding with that successful login

Sep 11 08:35:31 <hostname> sshd[25071]: error: Could not load host key: /etc/ssh/ssh_host_ed25519_key
Sep 11 08:35:32 <hostname> sshd[25071]: Accepted publickey for backups-only from <desktop IP> port 54242 ssh2: RSA e2:e6:07:27:c1:continues...
Sep 11 08:35:32 <hostname> sshd[25071]: pam_unix(sshd:session): session opened for user backups-only by (uid=0)
Sep 11 08:35:32 <hostname> systemd-logind[638]: New session 12 of user backups-only.
Sep 11 08:36:00 <hostname> sshd[25133]: Received disconnect from <desktop IP>: 11: disconnected by user
Sep 11 08:36:00 <hostname> sshd[25071]: pam_unix(sshd:session): session closed for user backups-only
16
  • If 3. works using the keyfile and 6. works also, then... err... what does sshd logfile on the receiving end say?
    – Jan
    Sep 7, 2014 at 13:20
  • @Jan I get Sep 7 14:45:01 <hostname> CRON[18716]: pam_unix(cron:session): session closed for user root Sep 7, 2014 at 13:47
  • That's either the wrong log line or the user trying to connect via ssh is root... Or is that from the machine that initiates the backups?
    – Jan
    Sep 7, 2014 at 13:48
  • 1
    Tom, 2 questions just to make sure In your first comment the logline has CRON[...], but it should look like Sep 7 16:06:02 <hostname> sshd[6747].... Are you 100% positive that this logline is from the server and that it is the correct line? The crontab you posted is the crontab of backups-only? Also, try to add the identity file manually: rsync .... -e 'ssh -i /home/user/.ssh/identity' ...
    – Jan
    Sep 10, 2014 at 21:40
  • 1
    Also, that line in auth.log you posted under Edit 2 is for cron running on the server, and should have nothing to do with your login attempts. Can you try tail -f /var/log/auth.log on the server while you're trying to run the script through cron? Also, I'm not sure if this would work, but can you try your first env command with rsync .... -e 'ssh -vvv -i /home/user/.ssh/identity ... to see if it spits out more errors?
    – Alaa Ali
    Sep 15, 2014 at 3:46

8 Answers 8

18
+300

Since everything is working fine from the command line, the error Permission denied (publickey) means that the SSH part of rsync is using a different identity file than the specified username.

From Jan's comment on the original question, we can specify the identity file in the rsync command using -e 'ssh -i /path/to/identity.file' ....

Using the below command to start off with a fresh environment in cron and specifying the complete path to the file apparently solves the issue:

env -i sh -c "rsync -lrstRO --delete --exclude 'lost+found' -e 'ssh -i /home/tom/.ssh/backups-only' /Backups/auto-daily-backups/./ [email protected]:/backups/desktop/"

I'm still really interested in this finding. It probably has to do with cron, the fact that it starts with minimal environment variables, and the ssh-agent. I'll be setting up the same scenario ina a couple of days to test it out and report back.

5
  • 2
    Do you mean you ran env -i sh -c "rsync -lrstRO --delete --exclude 'lost+found' -e 'ssh -i /path/to/identity.file' /Backups/auto-daily-backups/./ [email protected]:/backups/desktop/"
    – qazwsx
    Oct 9, 2014 at 23:49
  • @Problemania whops, fixed.
    – Alaa Ali
    Oct 10, 2014 at 0:15
  • I see that you have an answer, but I'm curious are you running 'sudo crontab -e' that is the root cron. What happens if you 'crontab -e' while logged in as the "backup" user.
    – wlraider70
    Oct 10, 2014 at 0:25
  • I think you meant this for the person that asked the question. But he was using his username's crontab, not root, and I think he didn't want to use the backup user's crontab.
    – Alaa Ali
    Oct 10, 2014 at 0:27
  • when I run a similar script with my user, it takes the ssh key via X11, so I needed a local copy if key, and make sure this file has the correct owner and rights, combined with above worked well for me.
    – Sverre
    Jun 30, 2015 at 4:28
1

I just have solved this problem that has kept me busy ..

Unable to connect in RSYNC over SSH, despite having stipulated the identity for SSH ...Nothing is done ... Rsync says "permission denied" and ssh tells me "read_passphrase: can not open /dev/tty: No device or address of this type"

But I read a post that explained that the crontab has its own environment that is not the same as root. I already knew that but I did not understand the impact it could have on SSH when using the SSH-AGENT

But my SSH key exchanges are done with PassPhrase ... so if the environment is different and my RSYNC over SSH expects a passphrase that can not be entered => SSH debug info also indicate the error:

"debug1: read_passphrase: can not open /dev/tty: No such device or address" => Well yes no TTY = no passphrase = not allowed

On my machine I use "Keychain" to have the SSH agent launched so I do not have to reenter the passphrase every time I try a remote connection. Keychain generates a file that contains the following information

SSH_AUTH_SOCK = /tmp/ssh-PWg3yHAARGmP/agent.18891; export SSH_AUTH_SOCK; SSH_AGENT_PID = 18893; export SSH_AGENT_PID;

==> The SSH-AGENT command returns the same info.

So, in the end, it is these information related to the current session that allow future authentications of the current session, without the need to enter the passphrase because already done previously and memorized ...

==> The solution is there ... it is enough in the script launched by the crontab, and to "source" the file containing this information or to do it on the command line ds the crontab ...

example: 14 09 * * *. /home/foo/.keychain/foo.serveur.org-sh && scp -vvv -P 22 /tmp/mon_fic/toto.sh [email protected] :. >> / var / log / check_connexion.log 2> & 1 or use the command "source /home/foo/keychain/foo.server.org-sh" in the script that starts a connection using SSH.

=> With this sourcing, no more worry. The information of SSH_AUTH_SOCK and SSH_AGENT_PID are loaded in the environment of the Crontab and are therefore known, the RSYNC over SSH works without any problem.

It has kept me busy but now, it works :)

1

Caveat for those who use SSH Agent Forwarding:

If you see this behaviour when debugging a script on a remote host, it's because even with the -e "ssh -i /path/to/key" flag, ssh will use your local (forwarded) key rather than the one on the server.

Concrete example: I have a script on the dev server that pulls in data from the "data server" using rsync over ssh. When I log into the dev server and run it, all is well, but when running from cron, I get the permission denied. Adding some verbosity to the SSH process (flag -vv) I noticed the following:

debug2: key: /home/nighty/.ssh/id_rsa (0x562d8b974820),
debug2: key: /home/juanr/.ssh/id_rsa (0x562d8b962930), explicit
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /home/nighty/.ssh/id_rsa
debug2: we sent a publickey packet, wait for reply
debug1: Server accepts key: pkalg ssh-rsa blen 279
debug2: input_userauth_pk_ok: fp 1a:19:08:9f:80:16:b1:db:55:42:9a:52:b2:49:9b:0a
debug1: Authentication succeeded (publickey).

What clued me off here is that by pure chance, I happen to have a different username on the local host ("nighty") than on the dev server ("juanr").

Notice how it marks the key on the dev server as "explicit", but still uses the forwarded key from my laptop to log in. Doing an ssh-copy-id at this point resolves nothing, because it simply reinstalls the forwarded key rather than the one from the dev server. If you use ssh-copy-id with agent forwarding, you need to specify which key to install with the -i flag: ssh-copy-id -i ~/.ssh/id_rsa.pub user@host.

1

A little late to the party, but I had the same issue. Local user crontab -l showed my daily backup task:

  0   5   *    *    5    /repos/local-bin/backup-backups daily-6

Worked when I called /repos/local-bin/backup-backups daily-6 directly. Worked when I called the underlying rsync command by itself. Worked when I ssh-ed into the systems I was attempting to back up. ssh-add -l showed my environment's agent/keychain had the keys I needed stored properly and readily available.

So, following the advice of the main answer, I added env to /repos/local-bin/backup-backups, made the cron task run immediately, copied the output from env, added export before each of the lines, opened up a new terminal, ran env -i sh, and pasted the export commands into it. Then I ran my ssh command from earlier and it failed. Finally! I noticed that the cron environment was receiving a pid for an ssh agent, though, so it should be working...

I ran ssh-add -l, saw that the latest SSH keys I created weren't added to this environment's agent/keychain... but some of my other keys were?? Which is weird, because it was working with the agent I was using outside of this environment... oh wait! I'm using KDE's wallet/agent outside of this (my desktop) environment, and using ssh-agent directly inside of this environment (my terminal environment)! So I pressed Ctrl+Alt+F2 to drop out of KDE, ran ssh-add -l, saw that it, too, was missing the keys I needed, and ran my add-ssh-keys script. Normally this is done automatically around boot/login time, but these were newly minted SSH keys that needed to be added to both my KDE keychain (which I had done properly) and my non-KDE ssh-agent (which I forgot to do). Hopefully this helps someone else avoid a few hours of hair pulling :)

0

Have you already tried the old trick of cleaning up the hosts files? I mean:

rm ~/.ssh/known_hosts

It's worth trying as ssh will rebuild it and you will get rid of stale stuff. You can of course also remove the parts belonging to a given IP / Host.

More questions: Is your cron job running under your UID or is it running as user cron or root?

3
  • 1
    The commands each work individually, so I don't see how deleting ~/.ssh/known_hosts would change anything? And cron runs as my user 'tom' on the desktop, with the intention of logging in to the server as user 'backups-only' with that corresponding (passwordless) SSH key, which is in user tom's ~/.ssh. Sep 14, 2014 at 11:03
  • 3
    @runlevel0 Neither the -r nor the -f flag is needed to delete known_hosts--it's a regular file (not a directory), and it is not readonly. rm .ssh/known-hosts would be considerably safer, considering that a single-character typo--accidentally adding a space between . and ssh/known_hosts after rm -rf (or rm -r) would usually delete the entire contents of the user's home folder! Sep 15, 2014 at 12:52
  • Hi Eliah, excellent point indeed!! I use the -rf flag as a reflex action, but you are absolutely right. Me bad.
    – runlevel0
    Sep 17, 2014 at 8:02
0

Use the rrsync script together with a dedicated ssh key as follows:

REMOTE server

mkdir ~/bin
gunzip /usr/share/doc/rsync/scripts/rrsync.gz -c > ~/bin/rrsync
chmod +x ~/bin/rrsync

LOCAL computer

ssh-keygen -f ~/.ssh/id_remote_backup -C "Automated remote backup"      #NO passphrase
scp ~/.ssh/id_remote_backup.pub [email protected]:/home/devel/.ssh

REMOTE computer

cat id_remote_backup.pub >> authorized_keys

Prepend to the newly added line the following

command="$HOME/bin/rrsync -ro ~/backups/",no-agent-forwarding,no-port-forwarding,no-pty,no-user-rc,no-X11-forwarding

So that the result looks like

command="$HOME/bin/rrsync -ro ~/backups/",no-agent-forwarding,no-port-forwarding,no-pty,no-user-rc,no-X11-forwarding ssh-rsa AAA...vp Automated remote backup

LOCAL

Put in your crontab the following script with x permission:

#!/bin/sh
echo ""
echo ""
echo "CRON:" `date`
set -xv
rsync -e "ssh -i $HOME/.ssh/id_remote_backup" -avzP [email protected]:/ /home/user/servidor 

Source: http://www.guyrutenberg.com/2014/01/14/restricting-ssh-access-to-rsync/

0

To try and debug add to the ssh part "ssh -v" this way you can get verbose mode with some helpful information.

Edit: From the man page:

-v      Verbose mode.  Causes ssh to print debugging messages about its progress.  This is helpful in debugging connection,
             authentication, and configuration problems.  Multiple -v options increase the verbosity.  The maximum is 3.
-1

I think you haven't configured the sshd_config file properly. Verify that PermitRootLogin yes and PubkeyAuthentication yes for remote maintenance.

2
  • 1
    He's not trying to login as root, and he probably has public key authentication setup correctly because he can ssh and even run the backup command from the terminal successfully.
    – Alaa Ali
    Sep 15, 2014 at 13:42
  • 1
    Thanks for the advice but I definitely do not have PermitRootLogin enabled and have no plans to change that. Best practice is to disable it, and ssh only as a normal user (add them to your 'sudoers' if necessary) and never as root. Sep 15, 2014 at 17:05

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .