6

I am currently trying to take a file (an image file such as test1.jpg) and I need to have a list of all duplicates of that file (by content). I've tried fdupes but that does not allow an input file to base its checks around.

TLDR: I need a way to list all duplicates of a specific file by their contents.

Preferable looking for a solution via the command-line, but full applications will be fine as well.

  • 2
    New to the site, update the tags as necessary please. – GamrCorps Oct 10 '16 at 22:01
  • Possible duplicate of How to find (and delete) duplicate files – Gilles Oct 11 '16 at 17:00
  • @Gilles the difference between these two questions is that this one is comparing everything to a reference file while the other question finds ALL duplicates – GamrCorps Oct 11 '16 at 17:02
8

First find the md5 hash of your file:

$ md5sum path/to/file
e740926ec3fce151a68abfbdac3787aa  path/to/file

(the first line is the command you need to execute, the second line is the md5 hash of that file)

Then copy the hash (it would be different in your case) and paste it into the next command:

$ find . -type f -print0 | xargs -0 md5sum | grep e740926ec3fce151a68abfbdac3787aa
e740926ec3fce151a68abfbdac3787aa  ./path/to/file
e740926ec3fce151a68abfbdac3787aa  ./path/to/other/file/with/same/content
....

If you want to get fancy you could combine the 2 in a single command:

$ find . -type f -print0 | xargs -0 md5sum | grep `md5sum path/to/file | cut -d " " -f 1`
e740926ec3fce151a68abfbdac3787aa  ./path/to/file
e740926ec3fce151a68abfbdac3787aa  ./path/to/other/file/with/same/content
....

You could use sha1 or any of the other fancy hashes if you want.

Edit

If the use case is to search through "several multi-gigabyte MP4s or iso-files" to find a "4 KB jpg" (as per @Tijn answer) then specifying the file size would speed things up dramatically.

If the size of the file you are looking for is exactly 3952 bytes (you can see that using ls -l path/to/file then this command would perform much faster:

$ find . -type f -size 3952c -print0 | xargs -0 md5sum | grep e740926ec3fce151a68abfbdac3787aa
e740926ec3fce151a68abfbdac3787aa  ./path/to/file
e740926ec3fce151a68abfbdac3787aa  ./path/to/other/file/with/same/content

Note the extra c after the size, indicating characters/bytes.

  • As this is not security related you could go for an even faster hash like the crc32 used in cksum. SHA1 is only going to be slower with no added benefit. – Calimo Oct 11 '16 at 13:10
  • crc32 is likely to have several duplicates if you have many files, and the processing cost is negligible compared to reading the files. – Simon Richter Oct 11 '16 at 16:43
4

Use diff command with boolean operators && and ||

bash-4.3$ diff /etc/passwd passwd_duplicate.txt > /dev/null && echo "SAME CONTENT" || echo "CONTENT DIFFERS"
SAME CONTENT

bash-4.3$ diff /etc/passwd TESTFILE.txt > /dev/null && echo "SAME CONTENT" || echo "CONTENT DIFFERS"
CONTENT DIFFERS

If you want to go over multiple files in specific directory, cd there and use a for loop like so:

bash-4.3$ for file in * ; do  diff /etc/passwd "$file" > /dev/null && echo "$file has same contents" || echo "$file has different contents"; done
also-waste.txt has different contents
directory_cleaner.py has different contents
dontdeletethisfile.txt has different contents
dont-delete.txt has different contents
important.txt has different contents
list.txt has different contents
neverdeletethis.txt has different contents
never-used-it.txt has different contents
passwd_dulicate.txt has same contents

For recursive cases, use find command to traverse directory and all its subdirectories(mind the quotes and all the appropriate slashes):

bash-4.3$ find . -type f -exec sh -c 'diff /etc/passwd "{}" > /dev/null &&  echo "{} same" || echo "{} differs"' \;
./reallyimportantfile.txt differs
./dont-delete.txt differs
./directory_cleaner.py differs
./TESTFILE.txt differs
./dontdeletethisfile.txt differs
./neverdeletethis.txt differs
./important.txt differs
./passwd_dulicate.txt same
./this-can-be-deleted.txt differs
./also-waste.txt differs
./never-used-it.txt differs
./list.txt differs
4

You can use filecmp in Python

For example:

import filecmp 
print filecmp.cmp('filename.png', 'filename.png') 

Will print True if equals, otherwise False

2

Get the md5sum of the file in question, and save in a variable e.g. md5:

md5=$(md5sum file.txt | awk '{print $1}')

Use find to traverse the desired directory tree, and check if any file has the same hash value, if so print the file name:

find . -type f -exec sh -c '[ "$(md5sum "$1" | awk "{print \$1}")" = "$2" ] \
                             && echo "$1"' _ {} "$md5" \;
  • find . -type f finds all files in the current directory, change the directory to meet your need

  • the -exec predicate executes the command sh -c ... on all files found

  • In sh -c, _ is a placeholder for $0, $1 is the file found, $2 is $md5

  • [ $(md5sum "$1"|awk "{print \$1}") = "$2" ] && echo "$1" prints the filename if the hash value of the file is same as the one we are checking duplicates for

Example:

% md5sum ../foo.txt bar.txt 
d41d8cd98f00b204e9800998ecf8427e  ../foo.txt
d41d8cd98f00b204e9800998ecf8427e  bar.txt

% md5=$(md5sum ../foo.txt | awk '{print $1}')

% find . -type f -exec sh -c '[ "$(md5sum "$1" | awk "{print \$1}")" = "$2" ] && echo "$1"' _ {} "$md5" \;
bar.txt
1

@smurf and @heemayl are certainly correct but I found out that in my case it was slower than I wanted it to be; I simply had too many files to process. Therefore I wrote a small command line tool that I think might help you too. (https://github.com/tijn/dupfinder ; ruby; no external dependencies)

Basically, my script postpones the hash calculation: it will only perform the calculation when file sizes are matching. Since why would I want to stream the contents of several multi-gigabyte MP4s or iso-files through a hash algorithm when I know I'm searching for a 4 KB jpg!? The rest of the script is mostly output formatting.

Edit: (thank you @Serg) Here's the source code of the whole script. You should save it in ~/bin/find-dups or maybe even /usr/local/bin/find-dups and then use chmod +x on it to make it executable. It needs to have Ruby installed but otherwise there are no other dependencies.

#!/usr/bin/env ruby

require 'digest/md5'
require 'fileutils'
require 'optparse'

def glob_from_argument(arg)
  if File.directory?(arg)
    arg + '/**/*'
  elsif File.file?(arg)
    arg
  else # it's already a glob
    arg
  end
end

# Wrap text at 80 chars. (configurable)
def wrap_text(*args)
  width = args.last.is_a?(Integer) ? args.pop : 80
  words = args.flatten.join(' ').split(' ')
  if words.any? { |word| word.size > width }
    raise NotImplementedError, 'cannot deal with long words'
  end

  lines = []
  line = []
  until words.empty?
    word = words.first
    if line.size + line.map(&:size).inject(0, :+) + word.size > width
      lines << line.join(' ')
      line = []
    else
      line << words.shift
    end
  end
  lines << line.join(' ') unless line.empty?
  lines.join("\n")
end

ALLOWED_PRINT_OPTIONS = %w(hay needle separator)

def parse_options(args)
  options = {}
  options[:print] = %w(hay needle)

  opt_parser = OptionParser.new do |opts|
    opts.banner = "Usage: #{$0} [options] HAYSTACK NEEDLES"
    opts.separator ''
    opts.separator 'Search for duplicate files (needles) in a directory (the haystack).'
    opts.separator ''
    opts.separator 'HAYSTACK should be the directory (or one file) that you want to search in.'
    opts.separator ''
    opts.separator wrap_text(
      'NEEDLES are the files you want to search for.',
      'A NEEDLE can be a file or a directory,',
      'in which case it will be recursively scanned.',
      'Note that NEEDLES may overlap the HAYSTACK.')
    opts.separator ''

    opts.on("-p", "--print PROPERTIES", Array,
      "When a match is found, print needle, or",
      "hay, or both. PROPERTIES is a comma-",
      "separated list with one or more of the",
      "words 'needle', 'hay', or 'separator'.",
      "'separator' prints an empty line.",
      "Default: 'needle,hay'") do |list|
      options[:print] = list
    end

    opts.on("-v", "--[no-]verbose", "Run verbosely") do |v|
      options[:verbose] = v
    end

    opts.on_tail("-h", "--help", "Show this message") do
      puts opts
      exit
    end
  end
  opt_parser.parse!(args)

  options[:haystack] = ARGV.shift
  options[:needles] = ARGV.shift(ARGV.size)

  raise ArgumentError, "Missing HAYSTACK" if options[:haystack].nil?
  raise ArgumentError, "Missing NEEDLES" if options[:needles].empty?
  unless options[:print].all? { |option| ALLOWED_PRINT_OPTIONS.include? option }
    raise ArgumentError, "Allowed print options are  'needle', 'hay', 'separator'"
  end

  options
rescue OptionParser::InvalidOption, ArgumentError => error
  puts error, nil, opt_parser.banner
  exit 1
end

options = parse_options(ARGV)

VERBOSE = options[:verbose]
PRINT_HAY = options[:print].include? 'hay'
PRINT_NEEDLE = options[:print].include? 'needle'
PRINT_SEPARATOR = options[:print].include? 'separator'

HAYSTACK_GLOB = glob_from_argument options[:haystack]
NEEDLES_GLOB = options[:needles].map { |arg| glob_from_argument(arg) }

def info(*strings)
  return unless VERBOSE
  STDERR.puts strings
end

def info_with_ellips(string)
  return unless VERBOSE
  STDERR.print string + '... '
end

def all_files(*globs)
  globs
    .map { |glob| Dir.glob(glob) }
    .flatten
    .map { |filename| File.expand_path(filename) } # normalize filenames
    .uniq
    .sort
    .select { |filename| File.file?(filename) }
end

def index_haystack(glob)
  all_files(glob).group_by { |filename| File.size(filename) }
end

@checksums = {}
def checksum(filename)
  @checksums[filename] ||= calculate_checksum(filename)
end

def calculate_checksum(filename)
  Digest::MD5.hexdigest(File.read(filename))
end

def find_needle(needle, haystack)
  straws = haystack[File.size(needle)] || return

  checksum_needle = calculate_checksum(needle)
  straws.detect do |straw|
    straw != needle &&
      checksum(straw) == checksum_needle &&
      FileUtils.identical?(needle, straw)
  end
end

BOLD = "\033[1m"
NORMAL = "\033[22m"

def print_found(needle, found)
  if PRINT_NEEDLE
    print BOLD if $stdout.tty?
    puts needle
    print NORMAL if $stdout.tty?
  end
  puts found if PRINT_HAY
  puts if PRINT_SEPARATOR
end

info "Searching #{HAYSTACK_GLOB} for files equal to #{NEEDLES_GLOB}."

info_with_ellips 'Indexing haystack by file size'
haystack = index_haystack(HAYSTACK_GLOB)
haystack[0] = nil # ignore empty files
info "#{haystack.size} files"

info 'Comparing...'
all_files(*NEEDLES_GLOB).each do |needle|
  info "  examining #{needle}"
  found = find_needle(needle, haystack)
  print_found(needle, found) unless found.nil?
end
  • 1
    While, github is well-respected and trusted site, It's recommended to put source code into the answers so that the answer can be self-sufficient. Refer to this discussion please : meta.askubuntu.com/q/15743/295286 – Sergiy Kolodyazhnyy Oct 11 '16 at 10:11
  • Seems a bit more complex but why to stop at this. You could even check the first X character to remove a percentage of file (mostly if you are looking for a big file). – AxelH Oct 11 '16 at 10:55
  • @AxelH Yes indeed, and I had some code doing exactly that but it turned out to be slower than without this functionality. I agree that it can be very useful though, for big files, so maybe I should add it back and optionally turn it on with some command line flag. (if anyone else wants to write it, a pull requests would be awesome) Now that I think about it, you could even choose which strategies to use an in which order. :-) – Tijn Oct 11 '16 at 11:35
1

It is possible to use the -c option of md5sum on the command line, if you do a little manipulation of its input stream. The following command is not recursive, it will only work in the present working directory. Replace original_file with the filename you wish to check duplicates against.

(hash=$(md5sum original_file) ; for f in ./* ; do echo "${hash%% *} ${f}" | if md5sum -c --status 2>/dev/null ; then echo "$f is a duplicate" ; fi ; done)

You can replace the for f in ./* part with for f in /directory/path/* to search a different directory.

If you would like the search to recurse through directories you can set the shell option 'globstar', and use two stars in the pattern given to the for loop:

(shopt -s globstar; hash=$(md5sum original_file); for f in ./** ; do echo "${hash%% *} ${f}" | if md5sum -c --status 2>/dev/null; then echo "$f is a duplicate"; fi; done)

Both version of the command will only output the name of duplicate files with the statement ./file is a duplicate. They're both encapsulated in brackets to avoid setting the hash variable or the globstar shell option outside of the command itself. The command can use other hashing algorithms such as sha256sum, just replace the two occurrences of md5sum to achieve this.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.