Rsync with special character files not working between Mac and Linux

Question

I want to use rsync to backup my Ubuntu server with a disk residing on a Mac. But I do not get it to work properly, since every time I re-run the rsync operation after the initial time, the files with special characters are first deleted and then re-synced. It seems as if there is a problem with different character sets.

The preferred solution seems to be to use the --iconv option:

You can use rsync's --iconv option to convert between UTF-8 NFC & NFD, at least if you're on a Mac. There is a special utf-8-mac character set that stands for UTF-8 NFD. So to copy files from your Mac to your NAS, you'd need to run something like:

rsync -a --iconv=utf-8-mac,utf-8 localdir/ mynas:remotedir/

This will convert all the local filenames from UTF-8 NFD to UTF-8 NFC on the remote server. The files' contents won't be affected.

Thanks to @Jan, I updated my rsync version on my mac from 2.6.9. to 3.1.1. Still, I am still not quite there, since I now receive a further error:

iconv_open("UTF-8", "utf-8-mac") failed
rsync error: requested action not supported (code 4) at rsync.c(118) [sender=3.0.9]
rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]
rsync error: error in rsync protocol data stream (code 12) at io.c(226) [Receiver=3.1.1]

I am at loss to understand why "requested action not supported" since it seems that the rsync version on my Ubuntu (12.04) is post 3.x.x, and thus should support the --iconv option.

EDIT: Let me add that when I (on the Mac, nota bene) initiate a rsync FROM the Mac TO Linux, everything works splendidly:

rsync -av --delete --iconv=utf-8-mac,utf-8 localdir/ mynas:remotedir/

But going the other way' from the mac does not work. Strangely enough, testing to initiate the rsync from the linux machine renders this strange message:

rsync: on remote machine: --iconv=UTF-8-MAC: unknown option
rsync error: syntax or usage error (code 1) at /SourceCache/rsync/rsync-45/rsync/main.c(1333) [server=2.6.9]
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(605) [sender=3.0.9]

including the very strange claim [server=2.6.9], although I have updated to 3.1.1 on the Mac. For some reasons, it looks as my linux machine 'sees' only the original rsync version on the Mac.

Any suggestion on how to solve this?

Answer 1

The solution was embarrassingly simple: Much due to a comment I read when researching the problem, I thought you were supposed to specify the character set in the order of transformation; but it seems as that is not the correct syntax. Rather, one should always use --iconv=utf-8-mac,utf-8 when initialising the rsync from the mac, and always use --iconv=utf-8,utf-8-mac when initialising the rsync from the linux machine, no matter if I want to sync files from the mac or linux machine.

Then it works like magic!

EDIT: Indeed, sometimes, checking the manual page closely is a good thing to do. Here it is, black on white:

--iconv=CONVERT_SPEC
              Rsync  can  convert  filenames between character sets using this
              option.  Using a CONVERT_SPEC of "." tells rsync to look up  the
              default  character-set via the locale setting.  Alternately, you
              can fully specify what conversion to do by giving a local and  a
              remote   charset   separated   by   a   comma   in   the   order
              --iconv=LOCAL,REMOTE, e.g.  --iconv=utf8,iso88591.   This  order
              ensures  that the option will stay the same whether you're push-
              ing  or  pulling  files.

Answer 2

I can confirm this works, I was having the same problem. In my case any file with an accented character was unreadable on the destination. I only spotted it by running a folder comparison on my Mac using the Compare Folders application: https://itunes.apple.com/gb/app/compare-folders/id816042486?mt=12

Added the above --iconv=utf-8-mac,utf-8 and BOOM! rsync replaced every accented file with a new one.

To add some information as it seems the above links no longer work, to upgrade rsync to 3.1.2, install Macports and run: sudo port install rsync

The reason why you were seeing the remote server returning version 2.6.9 is because the old version is actually still there and the remote server is seeing that one instead of the new one.

Version 2.6.9 is located in /usr/bin