Take the 2-minute tour ×
Ask Ubuntu is a question and answer site for Ubuntu users and developers. It's 100% free, no registration required.

I have two directories, they contain common files. I want to know if one directory contains the same file as the other. I found a script on the net but I want to need improve it to do recursively.

  #!/bin/bash

  # cmp_dir - program to compare two directories

  # Check for required arguments
  if [ $# -ne 2 ]; then
      echo "usage: $0 directory_1 directory_2" 1>&2
      exit 1
  fi

  # Make sure both arguments are directories
  if [ ! -d $1 ]; then
      echo "$1 is not a directory!" 1>&2
      exit 1
  fi

  if [ ! -d $2 ]; then
      echo "$2 is not a directory!" 1>&2
      exit 1
  fi

  # Process each file in directory_1, comparing it to directory_2
  missing=0
  for filename in $1/*; do
      fn=$(basename "$filename")
      if [ -f "$filename" ]; then
          if [ ! -f "$2/$fn" ]; then
              echo "$fn is missing from $2"
              missing=$((missing + 1))
          fi
      fi
  done
  echo "$missing files missing"

Would anybody suggest an algorithm for it?

share|improve this question

2 Answers 2

up vote 5 down vote accepted
#!/bin/bash

# cmp_dir - program to compare two directories

# Check for required arguments
if [ $# -ne 2 ]; then
  echo "usage: $0 directory_1 directory_2" 1>&2
  exit 1
fi

# Make sure both arguments are directories
if [ ! -d "$1" ]; then
  echo "$1 is not a directory!" 1>&2
  exit 1
fi

if [ ! -d "$2" ]; then
  echo "$2 is not a directory!" 1>&2
  exit 1
fi

# Process each file in directory_1, comparing it to directory_2
missing=0
while IFS= read -r -d $'\0' filename
do
  fn=${filename#$1}
  if [ ! -f "$2/$fn" ]; then
      echo "$fn is missing from $2"
      missing=$((missing + 1))
  fi
done < <(find "$1" -type f -print0)

echo "$missing files missing"

Note that I have added double-quotes around $1 and $2 at various places above to protect them shell expansion. Without the double-quotes, directory names with spaces or other difficult characters would cause errors.

The key loop now reads:

while IFS= read -r -d $'\0' filename
do
  fn=${filename#$1}
  if [ ! -f "$2/$fn" ]; then
      echo "$fn is missing from $2"
      missing=$((missing + 1))
  fi
done < <(find "$1" -type f -print0)

This uses find to recursively dive into directory $1 and find file names. The construction while IFS= read -r -d $'\0' filename; do .... done < <(find "$1" -type f -print0) is safe against all file names.

basename is no longer used because we are looking at files within subdirectories and we need to keep the subdirectories. So, in place of the call to basename, the line fn=${filename#$1} is used. This just removes from filename the prefix containing directory $1.

Problem 2

Suppose that we match files by name but regardless of directory. In other words, if the first directory contains a file a/b/c/some.txt, we will consider it present in the second directory if file some.txt exists in any subdirectory of the second directory. To do this replace the loop above with:

while IFS= read -r -d $'\0' filename
do
  fn=$(basename "$filename")
  if ! find "$2" -name "$fn" | grep -q . ; then
      echo "$fn is missing from $2"
      missing=$((missing + 1))
  fi
done < <(find "$1" -type f -print0)
share|improve this answer
    
Thank you for quick answer, I ll try your fix and post the result. –  kenn May 25 at 20:43
    
I prepared two directories like cmp1 and cmp2 I put a number of identical files in both of them, when I run above script, it reported the result as expected but when I move some files to a subfolder in cmp2 it reported moved files as missing in cmp2 –  kenn May 25 at 21:04
    
@kenn Yes. If you want it to search cmp2 for a matching file name anywhere within cmp2, that requires a different approach. –  John1024 May 25 at 21:24
    
@kenn I added a version to handle that case. –  John1024 May 25 at 21:40
    
Thank you very much John! I just checked out your last post today. It does what I want –  kenn May 26 at 8:56

FSlint is a small GUI application that helps you identify and clean your system of redundant files.

Installing FSlint

Install FSlint from the Ubuntu Software Center, or from the command line as follows:

sudo apt-get install fslint

(On my system, installing FSlint did not pull in additional dependencies. Specifically, fslint depends on findutils, python, and python-glade2, which should all be on your system already. You can remove FSlint using the Software Center or by typing sudo apt-get autoremove --purge fslint in a terminal).

Searching for Files

Launch FSlint from the Unity Dash.

Here is a screen-shot of the main screen. There are many advanced features, but basic usage of the application is relatively straightforward.

Click the Add button at the top left to add all the directories you would like to check. Obviously, you can remove directories using the Remove button.

enter image description here

Make sure the recurse? check-box at the the right is selected. Then click the Find button. (Any errors, such as file permission issues, will be printed at the bottom of the FSlint window).

FSlint will list all of the duplicate files, their directory locations, and the file date. FSlint also presents you with the number of bytes wasted due to the redundant files.

Removing Duplicates

Now you can select multiple files using the Shift or Ctrl keys and left mouse button. If you want to select multiple files automatically, click on the Select button and you will be given options such as selecting files based on date, or entering wild card selection criteria.

If you need to use the list of selected files outside of FSlint (perhaps as input to your own script) click on the Save button to save a text file.

Finally you can delete the selected files using the Delete button, or you can merge the selected files using the Merge button. Note that the Merge feature removes the unselected files from your system and creates hard links to the corresponding selected files. You would use this feature if you wanted to keep your existing file structure, but wanted to free up some space on your system.

enter image description here

Additional Features & Documentation

FSlint has other powerful features which are accessible from the tabs in the left pane. I have found Name clashes to be useful where there are files that have the same name, but are different (perhaps because you saved a newer version of a file in a different directory).

There is also an Advanced search parameters tab at the top of the FSlint window that allows you to exclude certain directories in your search, or filter your results using parameters.

There a re a lot of powerful features in this simple little tool. It may save you the effort of having to write and debug a script. You can find out more at http://www.pixelbeat.org/fslint/. Here is a direct link to the English guide: http://en.flossmanuals.net/fslint/.

share|improve this answer
    
As stated, you install it from the Software Center. –  PJ Singh May 26 at 12:22
    
Since this is a simple GUI application, it is pretty intuitive to use. Select a directory (or directories). Make sure the "Recursive" check box is selected. Then click the find button. The matching files are listed in groupings, showing how many bytes of redundant space they are using. You are then able to take an action, such as turning redundant files into links, or simply deleting them. I'm no at my computer right now so writ g this from my mobile Devi e is a bit cumbersome. Later I'll see if I can get to the PC and post some screen shots. –  PJ Singh May 26 at 13:05

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.