26

I've already copied terabytes of files with rsync but I forgot to use --archive to preserve files' special attributes.

I tried executing rsync again this time with --archive but it was way slower than what I expected. Is there any easy way to do this faster by just copying metadata recursively?

5

5 Answers 5

26

Ok, you can copy owner, group, permission and timestamps using the --reference parameter to chown, chmod, touch. Here is a script to do so

#!/bin/bash
# Filename: cp-metadata

myecho=echo
src_path="$1"
dst_path="$2"

find "$src_path" |
  while read src_file; do
    dst_file="$dst_path${src_file#$src_path}"
    $myecho chmod --reference="$src_file" "$dst_file"
    $myecho chown --reference="$src_file" "$dst_file"
    $myecho touch --reference="$src_file" "$dst_file"
  done

You should run it with sudo (to allow chown) and with two parameters: source and destination directory. The script only echo what it would do. If satisfied change the line myecho=echo with myecho=.

8
  • 1
    Yes, that's what I need: --reference in chmod. Thank you. And I really appreciate it if anyone could introduce something like chmod --reference for copying time-stamps.
    – Mohammad
    Aug 12, 2011 at 11:14
  • 1
    @Mohammad: for that you can use touch --reference=otherfile file. Updated the answer
    – enzotib
    Aug 12, 2011 at 11:18
  • That's great. Actually I was reading touch manual just now ;-)
    – Mohammad
    Aug 12, 2011 at 11:20
  • Just a note: touch by design only changes the modification and access times, the "creation" time is not affected. (I think ext2/3 do not support changing ctime anyway, but it might matter if you're using NTFS or the like).
    – Amro
    Apr 12, 2015 at 8:43
  • 1
    In case you want to only change metadata of existing files and don't need to assure the existence of files, add a -c switch to the touch command to stop it creating empty files in the $dst_path.
    – Synchro
    Jun 20, 2016 at 16:23
9

Treating the question as "rsync only has metadata to copy, so why is it so slow, and how can I make it faster?":

rsync usually uses equal mtimes as a heuristic to detect and skip unchanged files. Without --archive (specifically, without --times) the destination files' mtimes remain set to the time you rsync-ed them, while the source files' mtimes remain intact (ignoring manual trickery by you). Without external guarantees from you that the source files' contents haven't changed, rsync has to assume they might have and therefore has to checksum them and/or copy them to the destination again. This, plus the fact that --whole-file is implied for local->local syncs, makes rsync without --times approximately equivalent to cp for local syncs.

Provided that updating the destination files' contents is acceptable, or if the source files are untouched since the original copy, you should find rsync --archive --size-only quicker than a naive rsync.

If in doubt as to what rsync is copying that is taking so long, rsync --archive --dry-run --itemize-changes ... tells you in exhaustive, if terse, detail.

2
  • 4
    Very useful info. --archive --size-only is a great combo. Not only does it prevent recopying files that already exist in the destination, but it will also update their metadata. This was unexpected for me, because rsync's man page describes --size-only as "skipping" files whose sizes match. Turns out that it just skips the copy, but will still sync the metadata. Ideal. Mar 6, 2013 at 5:59
  • I did so, and it doesn't modify change or birth times (reported by stat). Aug 17, 2022 at 8:21
7

WARNING: Without special workarounds, GNU cp --attributes-only will truncate the destination files, at least in Precise. See the edit below.

Original:

In this situation you probably want GNU cp's --attributes-only option, together with --archive, as it's tried and tested code, does all filesystem-agnostic attributes and doesn't follow symlinks (following them can be bad!):

cp --archive --attributes-only /source/of/failed/backup/. /destination/

As with files, cp is additive with extended attributes: if both source and destination have extended attributes it adds the source's extended attributes to the destination (rather than deleting all of the destination's xattrs first). While this mirrors how cp behaves if you copy files into an existing tree, it might not be what you expect.

Also note that if you didn't preserve hard links the first time around with rsync but want to preserve them now then cp won't fix that for you; you're probably best off rerunning rsync with the right options (see my other answer) and being patient.

If you found this question while looking to deliberately separate and recombine metadata/file contents then you might want to take a look at metastore which is in the Ubuntu repositories.

Source: GNU coreutils manual


Edited to add:

cp from GNU coreutils >= 8.17 and above will work as described, but coreutils <= 8.16 will truncate files when restoring their metadata. If in doubt, don't use cp in this situation; use rsync with the right options and/or be patient.

I wouldn't recommend this unless you fully understand what you're doing, but earlier GNU cp can be prevented from truncating files using the LD_PRELOAD trick:

/*
 * File: no_trunc.c
 * Author: D.J. Capelis with minor changes by Zak Wilcox
 *
 * Compile:
 * gcc -fPIC -c -o no_trunc.o no_trunc.c
 * gcc -shared -o no_trunc.so no_trunc.o -ldl
 *
 * Use:
 * LD_PRELOAD="./no_trunc.so" cp --archive --attributes-only <src...> <dest>
 */

#define _GNU_SOURCE
#include <dlfcn.h>
#define _FCNTL_H
#include <bits/fcntl.h>

extern int errorno;

int (*_open)(const char *pathname, int flags, ...);
int (*_open64)(const char *pathname, int flags, ...);

int open(const char *pathname, int flags, mode_t mode) {
        _open = (int (*)(const char *pathname, int flags, ...)) dlsym(RTLD_NEXT, "open");
        flags &= ~(O_TRUNC);
        return _open(pathname, flags, mode);
}

int open64(const char *pathname, int flags, mode_t mode) {
        _open64 = (int (*)(const char *pathname, int flags, ...)) dlsym(RTLD_NEXT, "open64");
        flags &= ~(O_TRUNC);
        return _open64(pathname, flags, mode);
}
5
  • errorno should be errno, right?
    – enzotib
    Apr 14, 2015 at 7:41
  • A quick test removing it seems to work, so I guess I perpetuated a redundancy/mistake in the original, but everyone will be on newer coreutils by now anyway.
    – ZakW
    Apr 15, 2015 at 9:26
  • but what you call rsync with the right options is an answer to another question...
    – Jean Paul
    May 22, 2019 at 12:01
  • On MacOS, those attributes are not available. The command cp does not have --archive or --attributes-only. You can use cp -a sourcefile destfile. The -a will copy attributes, date, etc, but unfortunately copies the contents as well.
    – Sohail Si
    Jan 27, 2022 at 13:09
  • None of these work on MacOS. For a solution for MacOS, see my answer here: unix.stackexchange.com/questions/20645/…
    – Sohail Si
    Jan 27, 2022 at 16:40
3

I had to do this remotely to another computer so I couldn't use --reference

I used this to make the script...

find -printf "touch -d \"%Tc\" \"%P\"\n" >/tmp/touch.sh

But make sure there aren't any filenames with " in them first...

find | grep '"'

Then copy touch.sh to your remote computer, and run...

cd <DestinationFolder>; sh /tmp/touch.sh

There're also options in find -printf to print user,group name if you want copy those.

1
  • Thanks for the ideas to a) "just use a shell script" and b) to generate said script using find. I was in the same situation - forgot to copy attributes, source and destination disks were already in different machines and didn't really want to reverse that.
    – i336_
    Nov 8, 2018 at 12:32
2

In local transfers, when source and destination are on locally mounted filesystems, rsync will always copy whole files content. To avoid this you can use

rsync -a --no-whole-file source dest
2
  • I tried rsync with --no-whole-file and --progress and I can still see the copying progress (about 30 MB/s); so I guess it is not fast enough, yet. I am losing my hope on rsync...
    – Mohammad
    Aug 12, 2011 at 10:40
  • This option is used to tell rsync not to use the shortcut when files are both in local path, but it does not prevent rsync from copying the content.
    – Jean Paul
    May 22, 2019 at 11:45

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .