Comparing the contents of two directories

Question

I have two directories that should contain the same files and have the same directory structure.

I think that something is missing in one of these directories.

Using the bash shell, is there a way to compare my directories and see if one of them is missing files that are present in the other?

Answer 1

A good way to do this comparison is to use find with md5sum, then a diff.

Example

Use find to list all the files in the directory then calculate the md5 hash for each file and pipe it sorted by filename to a file:

find /dir1/ -type f -exec md5sum {} + | sort -k 2 > dir1.txt

Do the same procedure to the another directory:

find /dir2/ -type f -exec md5sum {} + | sort -k 2 > dir2.txt

Then compare the result two files with diff:

diff -u dir1.txt dir2.txt

Or as a single command using process substitution:

diff <(find /dir1/ -type f -exec md5sum {} + | sort -k 2) <(find /dir2/ -type f -exec md5sum {} + | sort -k 2)

This strategy is very useful when the two directories to be compared are not in the same machine and you need to make sure that the files are equal in both directories.

Another good way to do the job is using Git’s diff command:

git diff --no-index dir1/ dir2/

Answer 2

Maybe one option is to run rsync two times

rsync -r -n -t -v --progress -c -s /dir1/ /dir2/

With the previous line, you will get files that are in dir1 and are different (or missing) in dir2. Also folders with different date.

rsync -r -n -t -v --progress -c -s /dir2/ /dir1/

The same for dir2

#from the rsync --help :
-r, --recursive             recurse into directories
-n, --dry-run               perform a trial run with no changes made
-t, --times                 preserve modification times
-v, --verbose               increase verbosity
    --progress              show progress during transfer
-c, --checksum              skip based on checksum, not mod-time & size
-s, --protect-args          no space-splitting; only wildcard special-chars

You can delete the -n option to undergo the changes. That is copying the list of files to the second folder.

In case you do that, maybe a good option is to use -u, to avoid overwriting newer files.

-u, --update                skip files that are newer on the receiver

Answer 3

Fairly easy task to achieve in python:

python -c 'import os,sys;d1=os.listdir(sys.argv[1]);d2=os.listdir(sys.argv[2]);d1.sort();d2.sort();x="SAME" if d1 == d2 else "DIFF";print x' DIR1 DIR2

Substitute actual values for DIR1 and DIR2.

Here's sample run:

$ python -c 'import os,sys;d1=os.listdir(sys.argv[1]);d2=os.listdir(sys.argv[2]);d1.sort();d2.sort();x="SAME" if d1 == d2 else "DIFF";print x' Desktop/ Desktop
SAME
$ python -c 'import os,sys;d1=os.listdir(sys.argv[1]);d2=os.listdir(sys.argv[2]);d1.sort();d2.sort();x="SAME" if d1 == d2 else "DIFF";print x' Desktop/ Pictures/
DIFF

---

For readability, here's an actual script instead of one-liner:

#!/usr/bin/env python
import os, sys

d1 = os.listdir(sys.argv[1])
d2 = os.listdir(sys.argv[2])
d1.sort()
d2.sort()

if d1 == d2:
    print("SAME")
else:
    print("DIFF")

Answer 4

If you want to make each file expandable and collapsible, you can pipe the output of diff -r into Vim.

First let's give Vim a folding rule:

mkdir -p ~/.vim/ftplugin
echo "set foldexpr=getline(v:lnum)=~'^diff.*'?'>1':1 foldmethod=expr fdc=2" >> ~/.vim/ftplugin/diff.vim

Now just:

diff -r dir1 dir2 | vim -

You can hit zo and zc to open and close folds. To get out of Vim, hit :q<Enter>

Answer 5

Here is an alternative, to compare just filenames, and not their contents:

diff <(cd folder1 && find . | sort) <(cd folder2 && find . | sort)

This is an easy way to list missing files, but of course it won't detect files with the same name but different contents!

(Personally I use my own diffdirs script, but that is part of a larger library.)

Answer 6

Through you are not using bash, you can do it using diff with --brief and --recursive:

$ diff -rq dir1 dir2 
Only in dir2: file2
Only in dir1: file1

The man diff includes both options:

-q, --brief
report only when files differ

-r, --recursive
recursively compare any subdirectories found

Answer 7

You can use the diff command just as you would use it for files:

diff <directory1> <directory2>

If you want to see subfolders and -files too, you can use the -r option:

diff -r <directory1> <directory2>