Take the 2-minute tour ×
Ask Ubuntu is a question and answer site for Ubuntu users and developers. It's 100% free, no registration required.

I have a largish music collection and there are some duplicates in there. Is there any way to find duplicate files. At a minimum by doing a hash and seeing if two files have the same hash.

Bonus points for also finding files with the same name apart from the extension - I think I have some songs with both mp3 and ogg format versions.

I'm happy using the command line if that is the easiest way.

share|improve this question
    
Remove all the unwanted, duplicated files from your machine. Software name is DuplicateFilesDeleter. –  aidens Feb 26 at 15:43
add comment

7 Answers

up vote 31 down vote accepted

I use fdupes for this. It is a commandline program. You can call it like fdupes -r /dir/ect/ory and it will print out a list of dupes. fdupes has also a Wikipedia article, which lists some more programs.

share|improve this answer
1  
It also has a "-d" option that lets you choose which copy you want to keep, and deletes the other ones (or you can keep all of them if you want). –  Matthew Crumley Sep 8 '10 at 20:59
    
How can I use the -d option to fix my problem here –  John McKean Pruitt Apr 3 '12 at 22:07
    
Is it possible for fdupes to list duplicate folders instead of duplicate files? –  Anderson Green Sep 1 '12 at 22:31
    
No, I don't think so. –  qbi Oct 31 '12 at 20:46
    
great response! saved my life :) –  victorhqc Mar 21 '13 at 20:39
add comment

FSlint has a GUI and some other features. The explanation of the duplicate checking algorithm from their FAQ:

1. exclude files with unique lengths
2. handle files that are hardlinked to each other
3. exclude files with unique md5(first_4k(file))
4. exclude files with unique md5(whole file)
5. exclude files with unique sha1(whole file) (in case of md5 collisions).
share|improve this answer
3  
Thanks. Note that the command name is "fslint-gui", and the command line tools are not in $PATH by default - they are in /usr/share/fslint/fslint. I was confused when I didn't get help on which package it was in by just running fslint (via /usr/lib/command-not-found). –  nealmcb Nov 19 '11 at 18:59
add comment

List of programs/scripts/bash-solutions, that can find duplicates and run under nix:

dupedit - Compares many files at once without checksumming. Avoids comparing files against themselves when multiple paths point to the same file.

dupmerge - runs on various platforms (Win32/64 with Cygwin, *nix, Linux etc.)

dupseek - Perl with algorithm optimized to reduce reads.

fdf - Perl/c based and runs across most platforms (Win32, *nix and probably others). Uses MD5, SHA1 and other checksum algorithms

freedups - shell script, that searches through the directories you specify. When it finds two identical files, it hard links them together. Now the two or more files still exist in their respective directories, but only one copy of the data is stored on disk; both directory entries point to the same data blocks.

fslint - has command line interface and GUI.

liten - Pure Python deduplication command line tool, and library, using md5 checksums and a novel byte comparison algorithm. (Linux, Mac OS X, *nix, Windows)

liten2 - A rewrite of the original Liten, still a command line tool but with a faster interactive mode using SHA-1 checksums (Linux, Mac OS X, *nix)

rdfind - One of the few which rank duplicates based on the order of input parameters (directories to scan) in order not to delete in "original/well known" sources (if multiple directories are given). Uses MD5 or SHA1.

rmlint - Fast finder with command line interface and many options to find other lint too (uses MD5)

ua - Unix/Linux command line tool, designed to work with find (and the like).

findrepe - free Java-based command-line tool designed for an efficient search of duplicate files, it can search within zips and jars.(GNU/Linux, Mac OS X, *nix, Windows)

fdupe - a small script written in Perl. Doing its job fast and efficiently.1

ssdeep - identify almost identical files using Context Triggered Piecewise Hashing

share|improve this answer
1  
Are any of these programs able to find duplicate folders (not just duplicate files?) –  Anderson Green Sep 2 '12 at 3:08
add comment

If your deduplication task is music related, first run the picard application to correctly identify and tag your music (so that you find duplicate .mp3/.ogg files even if their names are incorrect). Note that picard is also available as an Ubuntu package.

That done, based on the musicip_puid tag you can easily find all your duplicate songs.

share|improve this answer
add comment

I use komparator (sudo apt-get install komparator) as GUI-tool for finding duplicates in manual mode. Hope this helps somebody.

share|improve this answer
add comment

Another script that does this job is rmdupe. From the author's page:

rmdupe uses standard linux commands to search within specified folders for duplicate files, regardless of filename or extension. Before duplicate candidates are removed they are compared byte-for-byte. rmdupe can also check duplicates against one or more reference folders, can trash files instead of removing them, allows for a custom removal command, and can limit its search to files of specified size. rmdupe includes a simulation mode which reports what will be done for a given command without actually removing any files.

share|improve this answer
add comment

For Music related duplicate identification and deletion Picard and Jaikoz by http://musicbrainz.org/ is the best solution. Jaikoz I believe automatically tags your music based on the data of the song file. You don't even need the name of the song for it to identify the song and assign all meta data to it. Although the free version can tag only a limited number of songs in one run, but you can run it as many times as you want.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.