I am currently trying to take a file (an image file such as test1.jpg) and I need to have a list of all duplicates of that file (by content). I've tried fdupes but that does not allow an input file to base its checks around.
TLDR: I need a way to list all duplicates of a specific file by their contents.
Preferable looking for a solution via the command-line, but full applications will be fine as well.
First find the md5 hash of your file:
$ md5sum path/to/file
e740926ec3fce151a68abfbdac3787aa path/to/file
(the first line is the command you need to execute, the second line is the md5 hash of that file)
Then copy the hash (it would be different in your case) and paste it into the next command:
$ find . -type f -print0 | xargs -0 md5sum | grep e740926ec3fce151a68abfbdac3787aa
e740926ec3fce151a68abfbdac3787aa ./path/to/file
e740926ec3fce151a68abfbdac3787aa ./path/to/other/file/with/same/content
....
If you want to get fancy you could combine the 2 in a single command:
$ find . -type f -print0 | xargs -0 md5sum | grep `md5sum path/to/file | cut -d " " -f 1`
e740926ec3fce151a68abfbdac3787aa ./path/to/file
e740926ec3fce151a68abfbdac3787aa ./path/to/other/file/with/same/content
....
You could use sha1 or any of the other fancy hashes if you want.
Edit
If the use case is to search through "several multi-gigabyte MP4s or iso-files" to find a "4 KB jpg" (as per @Tijn answer) then specifying the file size would speed things up dramatically.
If the size of the file you are looking for is exactly 3952 bytes (you can see that using ls -l path/to/file then this command would perform much faster:
$ find . -type f -size 3952c -print0 | xargs -0 md5sum | grep e740926ec3fce151a68abfbdac3787aa
e740926ec3fce151a68abfbdac3787aa ./path/to/file
e740926ec3fce151a68abfbdac3787aa ./path/to/other/file/with/same/content
Note the extra c after the size, indicating characters/bytes.
If you want you could combine this in a single command:
FILE=./path/to/file && find . -type f -size $(du -b $FILE | cut -f1)c -print0 | xargs -0 md5sum | grep $(md5sum $FILE | cut -f1 -d " ")
Use diff command with boolean operators && and ||
bash-4.3$ diff /etc/passwd passwd_duplicate.txt > /dev/null && echo "SAME CONTENT" || echo "CONTENT DIFFERS"
SAME CONTENT
bash-4.3$ diff /etc/passwd TESTFILE.txt > /dev/null && echo "SAME CONTENT" || echo "CONTENT DIFFERS"
CONTENT DIFFERS
If you want to go over multiple files in specific directory, cd there and use a for loop like so:
bash-4.3$ for file in * ; do diff /etc/passwd "$file" > /dev/null && echo "$file has same contents" || echo "$file has different contents"; done
also-waste.txt has different contents
directory_cleaner.py has different contents
dontdeletethisfile.txt has different contents
dont-delete.txt has different contents
important.txt has different contents
list.txt has different contents
neverdeletethis.txt has different contents
never-used-it.txt has different contents
passwd_dulicate.txt has same contents
For recursive cases, use find command to traverse directory and all its subdirectories(mind the quotes and all the appropriate slashes):
bash-4.3$ find . -type f -exec sh -c 'diff /etc/passwd "{}" > /dev/null && echo "{} same" || echo "{} differs"' \;
./reallyimportantfile.txt differs
./dont-delete.txt differs
./directory_cleaner.py differs
./TESTFILE.txt differs
./dontdeletethisfile.txt differs
./neverdeletethis.txt differs
./important.txt differs
./passwd_dulicate.txt same
./this-can-be-deleted.txt differs
./also-waste.txt differs
./never-used-it.txt differs
./list.txt differs
For example:
import filecmp
print filecmp.cmp('filename.png', 'filename.png')
Will print True if equals, otherwise False
Get the md5sum of the file in question, and save in a variable e.g. md5:
md5=$(md5sum file.txt | awk '{print $1}')
Use find to traverse the desired directory tree, and check if any file has the same hash value, if so print the file name:
find . -type f -exec sh -c '[ "$(md5sum "$1" | awk "{print \$1}")" = "$2" ] \
&& echo "$1"' _ {} "$md5" \;
find . -type f finds all files in the current directory, change the directory to meet your need
the -exec predicate executes the command sh -c ... on all files found
In sh -c, _ is a placeholder for $0, $1 is the file found, $2 is $md5
[ $(md5sum "$1"|awk "{print \$1}") = "$2" ] && echo "$1" prints the filename if the hash value of the file is same as the one we are checking duplicates for
Example:
% md5sum ../foo.txt bar.txt
d41d8cd98f00b204e9800998ecf8427e ../foo.txt
d41d8cd98f00b204e9800998ecf8427e bar.txt
% md5=$(md5sum ../foo.txt | awk '{print $1}')
% find . -type f -exec sh -c '[ "$(md5sum "$1" | awk "{print \$1}")" = "$2" ] && echo "$1"' _ {} "$md5" \;
bar.txt
It is possible to use the -c option of md5sum on the command line, if you do a little manipulation of its input stream. The following command is not recursive, it will only work in the present working directory. Replace original_file with the filename you wish to check duplicates against.
(hash=$(md5sum original_file) ; for f in ./* ; do echo "${hash%% *} ${f}" | if md5sum -c --status 2>/dev/null ; then echo "$f is a duplicate" ; fi ; done)
You can replace the for f in ./* part with for f in /directory/path/* to search a different directory.
If you would like the search to recurse through directories you can set the shell option 'globstar', and use two stars in the pattern given to the for loop:
(shopt -s globstar; hash=$(md5sum original_file); for f in ./** ; do echo "${hash%% *} ${f}" | if md5sum -c --status 2>/dev/null; then echo "$f is a duplicate"; fi; done)
Both version of the command will only output the name of duplicate files with the statement ./file is a duplicate. They're both encapsulated in brackets to avoid setting the hash variable or the globstar shell option outside of the command itself. The command can use other hashing algorithms such as sha256sum, just replace the two occurrences of md5sum to achieve this.
@smurf and @heemayl are certainly correct but I found out that in my case it was slower than I wanted it to be; I simply had too many files to process. Therefore I wrote a small command line tool that I think might help you too. (https://github.com/tijn/dupfinder ; ruby; no external dependencies)
Basically, my script postpones the hash calculation: it will only perform the calculation when file sizes are matching. Since why would I want to stream the contents of several multi-gigabyte MP4s or iso-files through a hash algorithm when I know I'm searching for a 4 KB jpg!? The rest of the script is mostly output formatting.
Edit: (thank you @Serg) Here's the source code of the whole script. You should save it in ~/bin/find-dups or maybe even /usr/local/bin/find-dups and then use chmod +x on it to make it executable. It needs to have Ruby installed but otherwise there are no other dependencies.
#!/usr/bin/env ruby
require 'digest/md5'
require 'fileutils'
require 'optparse'
def glob_from_argument(arg)
if File.directory?(arg)
arg + '/**/*'
elsif File.file?(arg)
arg
else # it's already a glob
arg
end
end
# Wrap text at 80 chars. (configurable)
def wrap_text(*args)
width = args.last.is_a?(Integer) ? args.pop : 80
words = args.flatten.join(' ').split(' ')
if words.any? { |word| word.size > width }
raise NotImplementedError, 'cannot deal with long words'
end
lines = []
line = []
until words.empty?
word = words.first
if line.size + line.map(&:size).inject(0, :+) + word.size > width
lines << line.join(' ')
line = []
else
line << words.shift
end
end
lines << line.join(' ') unless line.empty?
lines.join("\n")
end
ALLOWED_PRINT_OPTIONS = %w(hay needle separator)
def parse_options(args)
options = {}
options[:print] = %w(hay needle)
opt_parser = OptionParser.new do |opts|
opts.banner = "Usage: #{$0} [options] HAYSTACK NEEDLES"
opts.separator ''
opts.separator 'Search for duplicate files (needles) in a directory (the haystack).'
opts.separator ''
opts.separator 'HAYSTACK should be the directory (or one file) that you want to search in.'
opts.separator ''
opts.separator wrap_text(
'NEEDLES are the files you want to search for.',
'A NEEDLE can be a file or a directory,',
'in which case it will be recursively scanned.',
'Note that NEEDLES may overlap the HAYSTACK.')
opts.separator ''
opts.on("-p", "--print PROPERTIES", Array,
"When a match is found, print needle, or",
"hay, or both. PROPERTIES is a comma-",
"separated list with one or more of the",
"words 'needle', 'hay', or 'separator'.",
"'separator' prints an empty line.",
"Default: 'needle,hay'") do |list|
options[:print] = list
end
opts.on("-v", "--[no-]verbose", "Run verbosely") do |v|
options[:verbose] = v
end
opts.on_tail("-h", "--help", "Show this message") do
puts opts
exit
end
end
opt_parser.parse!(args)
options[:haystack] = ARGV.shift
options[:needles] = ARGV.shift(ARGV.size)
raise ArgumentError, "Missing HAYSTACK" if options[:haystack].nil?
raise ArgumentError, "Missing NEEDLES" if options[:needles].empty?
unless options[:print].all? { |option| ALLOWED_PRINT_OPTIONS.include? option }
raise ArgumentError, "Allowed print options are 'needle', 'hay', 'separator'"
end
options
rescue OptionParser::InvalidOption, ArgumentError => error
puts error, nil, opt_parser.banner
exit 1
end
options = parse_options(ARGV)
VERBOSE = options[:verbose]
PRINT_HAY = options[:print].include? 'hay'
PRINT_NEEDLE = options[:print].include? 'needle'
PRINT_SEPARATOR = options[:print].include? 'separator'
HAYSTACK_GLOB = glob_from_argument options[:haystack]
NEEDLES_GLOB = options[:needles].map { |arg| glob_from_argument(arg) }
def info(*strings)
return unless VERBOSE
STDERR.puts strings
end
def info_with_ellips(string)
return unless VERBOSE
STDERR.print string + '... '
end
def all_files(*globs)
globs
.map { |glob| Dir.glob(glob) }
.flatten
.map { |filename| File.expand_path(filename) } # normalize filenames
.uniq
.sort
.select { |filename| File.file?(filename) }
end
def index_haystack(glob)
all_files(glob).group_by { |filename| File.size(filename) }
end
@checksums = {}
def checksum(filename)
@checksums[filename] ||= calculate_checksum(filename)
end
def calculate_checksum(filename)
Digest::MD5.hexdigest(File.read(filename))
end
def find_needle(needle, haystack)
straws = haystack[File.size(needle)] || return
checksum_needle = calculate_checksum(needle)
straws.detect do |straw|
straw != needle &&
checksum(straw) == checksum_needle &&
FileUtils.identical?(needle, straw)
end
end
BOLD = "\033[1m"
NORMAL = "\033[22m"
def print_found(needle, found)
if PRINT_NEEDLE
print BOLD if $stdout.tty?
puts needle
print NORMAL if $stdout.tty?
end
puts found if PRINT_HAY
puts if PRINT_SEPARATOR
end
info "Searching #{HAYSTACK_GLOB} for files equal to #{NEEDLES_GLOB}."
info_with_ellips 'Indexing haystack by file size'
haystack = index_haystack(HAYSTACK_GLOB)
haystack[0] = nil # ignore empty files
info "#{haystack.size} files"
info 'Comparing...'
all_files(*NEEDLES_GLOB).each do |needle|
info " examining #{needle}"
found = find_needle(needle, haystack)
print_found(needle, found) unless found.nil?
end
