2

I have two directories, they contain common files. I want to know if one directory contains the same file as the other. I found a script on the net but I want to need improve it to do recursively.

  #!/bin/bash

  # cmp_dir - program to compare two directories

  # Check for required arguments
  if [ $# -ne 2 ]; then
      echo "usage: $0 directory_1 directory_2" 1>&2
      exit 1
  fi

  # Make sure both arguments are directories
  if [ ! -d $1 ]; then
      echo "$1 is not a directory!" 1>&2
      exit 1
  fi

  if [ ! -d $2 ]; then
      echo "$2 is not a directory!" 1>&2
      exit 1
  fi

  # Process each file in directory_1, comparing it to directory_2
  missing=0
  for filename in $1/*; do
      fn=$(basename "$filename")
      if [ -f "$filename" ]; then
          if [ ! -f "$2/$fn" ]; then
              echo "$fn is missing from $2"
              missing=$((missing + 1))
          fi
      fi
  done
  echo "$missing files missing"

Would anybody suggest an algorithm for it?

3 Answers 3

7
#!/bin/bash

# cmp_dir - program to compare two directories

# Check for required arguments
if [ $# -ne 2 ]; then
  echo "usage: $0 directory_1 directory_2" 1>&2
  exit 1
fi

# Make sure both arguments are directories
if [ ! -d "$1" ]; then
  echo "$1 is not a directory!" 1>&2
  exit 1
fi

if [ ! -d "$2" ]; then
  echo "$2 is not a directory!" 1>&2
  exit 1
fi

# Process each file in directory_1, comparing it to directory_2
missing=0
while IFS= read -r -d $'\0' filename
do
  fn=${filename#$1}
  if [ ! -f "$2/$fn" ]; then
      echo "$fn is missing from $2"
      missing=$((missing + 1))
  fi
done < <(find "$1" -type f -print0)

echo "$missing files missing"

Note that I have added double-quotes around $1 and $2 at various places above to protect them shell expansion. Without the double-quotes, directory names with spaces or other difficult characters would cause errors.

The key loop now reads:

while IFS= read -r -d $'\0' filename
do
  fn=${filename#$1}
  if [ ! -f "$2/$fn" ]; then
      echo "$fn is missing from $2"
      missing=$((missing + 1))
  fi
done < <(find "$1" -type f -print0)

This uses find to recursively dive into directory $1 and find file names. The construction while IFS= read -r -d $'\0' filename; do .... done < <(find "$1" -type f -print0) is safe against all file names.

basename is no longer used because we are looking at files within subdirectories and we need to keep the subdirectories. So, in place of the call to basename, the line fn=${filename#$1} is used. This just removes from filename the prefix containing directory $1.

Problem 2

Suppose that we match files by name but regardless of directory. In other words, if the first directory contains a file a/b/c/some.txt, we will consider it present in the second directory if file some.txt exists in any subdirectory of the second directory. To do this replace the loop above with:

while IFS= read -r -d $'\0' filename
do
  fn=$(basename "$filename")
  if ! find "$2" -name "$fn" | grep -q . ; then
      echo "$fn is missing from $2"
      missing=$((missing + 1))
  fi
done < <(find "$1" -type f -print0)
5
  • Thank you for quick answer, I ll try your fix and post the result.
    – kenn
    May 25, 2014 at 20:43
  • I prepared two directories like cmp1 and cmp2 I put a number of identical files in both of them, when I run above script, it reported the result as expected but when I move some files to a subfolder in cmp2 it reported moved files as missing in cmp2
    – kenn
    May 25, 2014 at 21:04
  • @kenn Yes. If you want it to search cmp2 for a matching file name anywhere within cmp2, that requires a different approach.
    – John1024
    May 25, 2014 at 21:24
  • @kenn I added a version to handle that case.
    – John1024
    May 25, 2014 at 21:40
  • Thank you very much John! I just checked out your last post today. It does what I want
    – kenn
    May 26, 2014 at 8:56
3

FSlint is a small GUI application that helps you identify and clean your system of redundant files.

Installing FSlint

Install FSlint from the Ubuntu Software Center, or from the command line as follows:

sudo apt-get install fslint

(On my system, installing FSlint did not pull in additional dependencies. Specifically, fslint depends on findutils, python, and python-glade2, which should all be on your system already. You can remove FSlint using the Software Center or by typing sudo apt-get autoremove --purge fslint in a terminal).

Searching for Files

Launch FSlint from the Unity Dash.

Here is a screen-shot of the main screen. There are many advanced features, but basic usage of the application is relatively straightforward.

Click the Add button at the top left to add all the directories you would like to check. Obviously, you can remove directories using the Remove button.

enter image description here

Make sure the recurse? check-box at the the right is selected. Then click the Find button. (Any errors, such as file permission issues, will be printed at the bottom of the FSlint window).

FSlint will list all of the duplicate files, their directory locations, and the file date. FSlint also presents you with the number of bytes wasted due to the redundant files.

Removing Duplicates

Now you can select multiple files using the Shift or Ctrl keys and left mouse button. If you want to select multiple files automatically, click on the Select button and you will be given options such as selecting files based on date, or entering wild card selection criteria.

If you need to use the list of selected files outside of FSlint (perhaps as input to your own script) click on the Save button to save a text file.

Finally you can delete the selected files using the Delete button, or you can merge the selected files using the Merge button. Note that the Merge feature removes the unselected files from your system and creates hard links to the corresponding selected files. You would use this feature if you wanted to keep your existing file structure, but wanted to free up some space on your system.

enter image description here

Additional Features & Documentation

FSlint has other powerful features which are accessible from the tabs in the left pane. I have found Name clashes to be useful where there are files that have the same name, but are different (perhaps because you saved a newer version of a file in a different directory).

There is also an Advanced search parameters tab at the top of the FSlint window that allows you to exclude certain directories in your search, or filter your results using parameters.

There a re a lot of powerful features in this simple little tool. It may save you the effort of having to write and debug a script. You can find out more at http://www.pixelbeat.org/fslint/. Here is a direct link to the English guide: http://en.flossmanuals.net/fslint/.

0
1

I want to share this way because I think it is quite fun.

For each subfolder below the target folder, we generate a hash for that subfolder.

Each folder hash is generated from the result of hashing all the files below that folder. So any folder containing identical files in the same structure should produce the same hash!

At the end, we use uniq to display only the duplicate folder hashes.

Save the following script as seek_duplicate_folders.sh and then run it like this:

$ bash seek_duplicate_folders.sh [root_folder_to_scan]

Here is the script:

#!/bin/bash
target="$1"

hash_folder() {
  echo "Hashing $1" >/dev/stderr
  pushd "$1" >/dev/null

  # Hash all the files
  find . -type f | sort | xargs md5sum |

  # Hash that list of hashes, discard the newline character,
  # and append the folder name
  md5sum - | tr -d '\n'
  printf "  %s\n" "$1"

  popd >/dev/null
}

find "$target" -type d |
while read dir
do hash_folder "$dir"
done |
sort |
# Display only the lines with duplicate hashes (first 32 chars are duplicates)
uniq -D -w 32

Caveats:

  • Inefficient: it md5sums files deep in the tree multiple times (once per ancestor folder)
  • Does not detect differences in file timestamps, ownership or permissions
  • Ignores empty folders. So two folders which contain identical files, or no files, but have different empty folders inside them, will still be reported as identical.
  • Ignores symlinks. Files containing different symlinks may still be reported as identical.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .