4

I have an old backup of documents. In my current Documents directory, a lot of these files exist in different locations with different names. I'm trying to find a way to show which files exist in the backup that do not exist in the Documents directory, preferably nice and GUI-y so that I can easily overview a lot of documents.

When I search for this question, a lot of people are looking for ways to do the opposite. There are tools like FSlint and DupeGuru, but they show duplicates. There is no invert mode.

4

If you are ready to use CLI, the following command should work for you:

diff --brief -r backup/ documents/

This will show you the files that are unique to each folder. If you want you can also ignore filename cases with the --ignore-file-name-case

As an example:

ron@ron:~/test$ ls backup/
file1  file2  file3  file4  file5
ron@ron:~/test$ ls documents/
file4  file5  file6  file7  file8
ron@ron:~/test$ diff backup/ documents/
Only in backup/: file1
Only in backup/: file2
Only in backup/: file3
Only in documents/: file6
Only in documents/: file7
Only in documents/: file8
ron@ron:~/test$ diff backup/ documents/ | grep "Only in backup"
Only in backup/: file1
Only in backup/: file2
Only in backup/: file3

In addition, if you want to report only when the files differ (and not report the actual 'difference'), you can use the --brief option as in:

ron@ron:~/test$ cat backup/file5 
one
ron@ron:~/test$ cat documents/file5
ron@ron:~/test$ diff --brief backup/ documents/
Only in backup/: file1
Only in backup/: file2
Only in backup/: file3
Files backup/file5 and documents/file5 differ
Only in documents/: file6
Only in documents/: file7
Only in documents/: file8

There are several visual diff tools such as meld that can do the same thing. You can install meld from the universe repository by:

sudo apt-get install meld

and use its "Directory comparison" option. Select the folder you want to compare. After selection you can compare them side-by-side:

enter image description here

fdupes is an excellent program to find the duplicate files but it does not list the non-duplicate files, which is what you are looking for. However, we can list the files that are not in the fdupes output using a combination of find and grep.

The following example lists the files that are unique to backup.

ron@ron:~$ tree backup/ documents/
backup/
├── crontab
├── dir1
│   └── du.txt
├── lo.txt
├── ls.txt
├── lu.txt
└── notes.txt
documents/
├── du.txt
├── lo-renamed.txt
├── ls.txt
└── lu.txt

1 directory, 10 files
ron@ron:~$ fdupes -r backup/ documents/ > dup.txt
ron@ron:~$ find backup/ -type f | grep -Fxvf dup.txt 
backup/crontab
backup/notes.txt
|improve this answer|||||
  • 1
    I love your elaborate response with pictures so you'll definitely get an upvote from me. And +1 for mentioning meld, an awesome tool especially for diffing source code. However, these tricks do not help in the case set out in the question; files existing in different locations with different names. There will need to be some hashing and matching involved. – Redsandro May 7 '16 at 11:06
  • 1
    @Redsandro Please see my updated answer. Does it work for you? – Ron May 12 '16 at 17:06
  • Hi @Ron, thank you for trying to find a solution that works. However, this is still way too much trouble for syncing thousands of files. I ended up deleting all duplicates so I was left with only uniques. I posted the workflow in an answer below. – Redsandro May 14 '16 at 12:33
1

I figured that the best workflow to merge old backups with thousands of files, archived under different directories with different names is to use DupeGuru after all. It looks a lot like the duplicates tab from FSlint, but it has the extra important feature of adding sources as 'reference'.

  1. Add your target directory (e.g. ~/Documents) as a reference.
    • A reference is read-only and no files will be removed
  2. Add your backup directory as normal.
  3. Find duplicates. Remove all duplicates that are found from the backup.
  4. You are left with only unique files in the backup directory. Use FreeFileSync or Meld to merge these, or merge manually.

If you have multiple old backup directories, it makes sense to merge the newest backup directory like this first, and then use this backup directory as a reference to clean it's duplicates from the older backups before merging them to the main document directory. This safes a lot of work where you don't have to remove unique files that you want to trash in stead of merge from the backups.

Remember to make a fresh backup after you've destroyed all old backups in the process. :)

|improve this answer|||||

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.