18

I make frequent backups to a local drive which I want to sync daily to a remote server.

The target server is configured for SSH key (no password) access only. Since my primary SSH key for that server is passphrase-protected, I've created a second SSH key (not passphrase protected) + user to use for unattended backups - this way I do not have to be present to enter my passphrase when cron runs.

I'm using cron and rsync, and all of the commands work individually, but fail when combined.

The furthest I've got while troubleshooting is running

env -i sh -c "rsync -lrstRO --delete --exclude 'lost+found' /Backups/auto-daily-backups/./ backups-only@XX.XX.XX.XX:/backups/desktop/"

which returns the error

Permission denied (publickey).
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: unexplained error (code 255) at io.c(226) [sender=3.1.0]

Any tips on how to troubleshoot this further?


Here's what I've tried so far and I'm out of ideas:

  1. Cron is definitely running ps aux | grep cron
  2. Nothing unusual in /var/log/syslog Sep 7 13:22:01 desktop CRON[6735]: (tom) CMD (sh /home/tom/Documents/Scripts/offsite-backup)

  3. SSH in Terminal to remote server as the backup user works ssh backups-user@XX.XX.XX.XX

  4. Running the command in Terminal works perfectly rsync -lrstRO --delete --exclude 'lost+found' /Backups/auto-daily-backups/./ backups-only@XX.XX.XX.XX:/backups/desktop/
  5. Manually specifying the path to the backups-user key has no effect rsync -lrstRO --delete --exclude 'lost+found' -e 'ssh -i /home/tom/.ssh/backups-only' /Backups/auto-daily-backups/./ backups-only@XX.XX.XX.XX:/backups/desktop/

  6. Replacing the non-functioning command with a simple test command works echo "Hello world" > ~/Desktop/test.txt

  7. Shouting/swearing at the computer had no effect (but made me feel better temporarily).


Edit 1:

Here's my crontab file and the script it calls.

...
# m h  dom mon dow   command
MAILTO=""
* * * * * sh /home/tom/Documents/Scripts/offsite-backup

and

#!/bin/bash

rsync -lrstRO --delete --exclude 'lost+found' /Backups/auto-daily-backups/./ backups-only@XX.XX.XX.XX:/backups/desktop/

Edit 2:

Just to clarify, /var/log/auth.log on the target server contains the line Sep 11 08:23:01 <hostname> CRON[24421]: pam_unix(cron:session): session closed for user root This is confusing because I'm no longer running cron every minute locally, but a new entry still appears every minute in the server logs. Crontab files for all users (including root) on the server are empty & do nothing.

Also, user 'backups-only' was created only on the server and with limited rights, with a dedicated SSH key copied to my desktop machine. I'm assuming this is the way to go because everything works when running the commands manually.

The crontab file posted above is for me, user 'tom' on my desktop machine. My intent is to have it call the script which should log in to the server as user 'backups-only'. I just tried running the backup script (rather than the command inside it) and it successfully connected & worked. I ran it on my desktop as user 'tom', same user who created the cron job that won't work. Here's the output from the server log corresponding with that successful login

Sep 11 08:35:31 <hostname> sshd[25071]: error: Could not load host key: /etc/ssh/ssh_host_ed25519_key
Sep 11 08:35:32 <hostname> sshd[25071]: Accepted publickey for backups-only from <desktop IP> port 54242 ssh2: RSA e2:e6:07:27:c1:continues...
Sep 11 08:35:32 <hostname> sshd[25071]: pam_unix(sshd:session): session opened for user backups-only by (uid=0)
Sep 11 08:35:32 <hostname> systemd-logind[638]: New session 12 of user backups-only.
Sep 11 08:36:00 <hostname> sshd[25133]: Received disconnect from <desktop IP>: 11: disconnected by user
Sep 11 08:36:00 <hostname> sshd[25071]: pam_unix(sshd:session): session closed for user backups-only
  • If 3. works using the keyfile and 6. works also, then... err... what does sshd logfile on the receiving end say? – Jan Sep 7 '14 at 13:20
  • @Jan I get Sep 7 14:45:01 <hostname> CRON[18716]: pam_unix(cron:session): session closed for user root – Tom Brossman Sep 7 '14 at 13:47
  • That's either the wrong log line or the user trying to connect via ssh is root... Or is that from the machine that initiates the backups? – Jan Sep 7 '14 at 13:48
  • 1
    Tom, 2 questions just to make sure In your first comment the logline has CRON[...], but it should look like Sep 7 16:06:02 <hostname> sshd[6747].... Are you 100% positive that this logline is from the server and that it is the correct line? The crontab you posted is the crontab of backups-only? Also, try to add the identity file manually: rsync .... -e 'ssh -i /home/user/.ssh/identity' ... – Jan Sep 10 '14 at 21:40
  • 1
    Also, that line in auth.log you posted under Edit 2 is for cron running on the server, and should have nothing to do with your login attempts. Can you try tail -f /var/log/auth.log on the server while you're trying to run the script through cron? Also, I'm not sure if this would work, but can you try your first env command with rsync .... -e 'ssh -vvv -i /home/user/.ssh/identity ... to see if it spits out more errors? – Alaa Ali Sep 15 '14 at 3:46
13
+300

Since everything is working fine from the command line, the error Permission denied (publickey) means that the SSH part of rsync is using a different identity file than the specified username.

From Jan's comment on the original question, we can specify the identity file in the rsync command using -e 'ssh -i /path/to/identity.file' ....

Using the below command to start off with a fresh environment in cron and specifying the complete path to the file apparently solves the issue:

env -i sh -c "rsync -lrstRO --delete --exclude 'lost+found' -e 'ssh -i /home/tom/.ssh/backups-only' /Backups/auto-daily-backups/./ backups-only@XX.XX.XX.XX:/backups/desktop/"

I'm still really interested in this finding. It probably has to do with cron, the fact that it starts with minimal environment variables, and the ssh-agent. I'll be setting up the same scenario ina a couple of days to test it out and report back.

  • 1
    Do you mean you ran env -i sh -c "rsync -lrstRO --delete --exclude 'lost+found' -e 'ssh -i /path/to/identity.file' /Backups/auto-daily-backups/./ backups-only@XX.XX.XX.XX:/backups/desktop/" – qazwsx Oct 9 '14 at 23:49
  • @Problemania whops, fixed. – Alaa Ali Oct 10 '14 at 0:15
  • I see that you have an answer, but I'm curious are you running 'sudo crontab -e' that is the root cron. What happens if you 'crontab -e' while logged in as the "backup" user. – wlraider70 Oct 10 '14 at 0:25
  • I think you meant this for the person that asked the question. But he was using his username's crontab, not root, and I think he didn't want to use the backup user's crontab. – Alaa Ali Oct 10 '14 at 0:27
  • when I run a similar script with my user, it takes the ssh key via X11, so I needed a local copy if key, and make sure this file has the correct owner and rights, combined with above worked well for me. – Sverre Jun 30 '15 at 4:28
0

Have you already tried the old trick of cleaning up the hosts files? I mean:

rm ~/.ssh/known_hosts

It's worth trying as ssh will rebuild it and you will get rid of stale stuff. You can of course also remove the parts belonging to a given IP / Host.

More questions: Is your cron job running under your UID or is it running as user cron or root?

  • 1
    The commands each work individually, so I don't see how deleting ~/.ssh/known_hosts would change anything? And cron runs as my user 'tom' on the desktop, with the intention of logging in to the server as user 'backups-only' with that corresponding (passwordless) SSH key, which is in user tom's ~/.ssh. – Tom Brossman Sep 14 '14 at 11:03
  • 3
    @runlevel0 Neither the -r nor the -f flag is needed to delete known_hosts--it's a regular file (not a directory), and it is not readonly. rm .ssh/known-hosts would be considerably safer, considering that a single-character typo--accidentally adding a space between . and ssh/known_hosts after rm -rf (or rm -r) would usually delete the entire contents of the user's home folder! – Eliah Kagan Sep 15 '14 at 12:52
  • Hi Eliah, excellent point indeed!! I use the -rf flag as a reflex action, but you are absolutely right. Me bad. – runlevel0 Sep 17 '14 at 8:02
0

Use the rrsync script together with a dedicated ssh key as follows:

REMOTE server

mkdir ~/bin
gunzip /usr/share/doc/rsync/scripts/rrsync.gz -c > ~/bin/rrsync
chmod +x ~/bin/rrsync

LOCAL computer

ssh-keygen -f ~/.ssh/id_remote_backup -C "Automated remote backup"      #NO passphrase
scp ~/.ssh/id_remote_backup.pub devel@10.10.10.83:/home/devel/.ssh

REMOTE computer

cat id_remote_backup.pub >> authorized_keys

Prepend to the newly added line the following

command="$HOME/bin/rrsync -ro ~/backups/",no-agent-forwarding,no-port-forwarding,no-pty,no-user-rc,no-X11-forwarding

So that the result looks like

command="$HOME/bin/rrsync -ro ~/backups/",no-agent-forwarding,no-port-forwarding,no-pty,no-user-rc,no-X11-forwarding ssh-rsa AAA...vp Automated remote backup

LOCAL

Put in your crontab the following script with x permission:

#!/bin/sh
echo ""
echo ""
echo "CRON:" `date`
set -xv
rsync -e "ssh -i $HOME/.ssh/id_remote_backup" -avzP devel@10.10.10.83:/ /home/user/servidor 

Source: http://www.guyrutenberg.com/2014/01/14/restricting-ssh-access-to-rsync/

0

To try and debug add to the ssh part "ssh -v" this way you can get verbose mode with some helpful information.

Edit: From the man page:

-v      Verbose mode.  Causes ssh to print debugging messages about its progress.  This is helpful in debugging connection,
             authentication, and configuration problems.  Multiple -v options increase the verbosity.  The maximum is 3.
0

I just have solved this problem that has kept me busy ..

Unable to connect in RSYNC over SSH, despite having stipulated the identity for SSH ...Nothing is done ... Rsync says "permission denied" and ssh tells me "read_passphrase: can not open /dev/tty: No device or address of this type"

But I read a post that explained that the crontab has its own environment that is not the same as root. I already knew that but I did not understand the impact it could have on SSH when using the SSH-AGENT

But my SSH key exchanges are done with PassPhrase ... so if the environment is different and my RSYNC over SSH expects a passphrase that can not be entered => SSH debug info also indicate the error:

"debug1: read_passphrase: can not open /dev/tty: No such device or address" => Well yes no TTY = no passphrase = not allowed

On my machine I use "Keychain" to have the SSH agent launched so I do not have to reenter the passphrase every time I try a remote connection. Keychain generates a file that contains the following information

SSH_AUTH_SOCK = /tmp/ssh-PWg3yHAARGmP/agent.18891; export SSH_AUTH_SOCK; SSH_AGENT_PID = 18893; export SSH_AGENT_PID;

==> The SSH-AGENT command returns the same info.

So, in the end, it is these information related to the current session that allow future authentications of the current session, without the need to enter the passphrase because already done previously and memorized ...

==> The solution is there ... it is enough in the script launched by the crontab, and to "source" the file containing this information or to do it on the command line ds the crontab ...

example: 14 09 * * *. /home/foo/.keychain/foo.serveur.org-sh && scp -vvv -P 22 /tmp/mon_fic/toto.sh foo@my-server.fr :. >> / var / log / check_connexion.log 2> & 1 or use the command "source /home/foo/keychain/foo.server.org-sh" in the script that starts a connection using SSH.

=> With this sourcing, no more worry. The information of SSH_AUTH_SOCK and SSH_AGENT_PID are loaded in the environment of the Crontab and are therefore known, the RSYNC over SSH works without any problem.

It has kept me busy but now, it works :)

New contributor
SoniXtrA is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.
-1

I think you haven't configured the sshd_config file properly. Verify that PermitRootLogin yes and PubkeyAuthentication yes for remote maintenance.

  • 1
    He's not trying to login as root, and he probably has public key authentication setup correctly because he can ssh and even run the backup command from the terminal successfully. – Alaa Ali Sep 15 '14 at 13:42
  • 1
    Thanks for the advice but I definitely do not have PermitRootLogin enabled and have no plans to change that. Best practice is to disable it, and ssh only as a normal user (add them to your 'sudoers' if necessary) and never as root. – Tom Brossman Sep 15 '14 at 17:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.