Ask Ubuntu is a question and answer site for Ubuntu users and developers. It only takes a minute to sign up.
Sign up to join this community
I have a largish music collection and there are some duplicates in there. Is there any way to find duplicate files. At a minimum by doing a hash and seeing if two files have the same hash.
Bonus points for also finding files with the same name apart from the extension - I think I have some songs with both mp3 and ogg format versions.
I'm happy using the command line if that is the easiest way.
I use fdupes for this. It is a commandline program which can be installed from the repositories with sudo apt install fdupes. You can call it like fdupes -r /dir/ect/ory and it will print out a list of dupes. fdupes has also a README on GitHub and a Wikipedia article, which lists some more programs.
fdupes -r . -d -N should save the first instance and delete the dupes. I just successfully cleared a single folder using fdupes . -d -N non recursively
List of programs/scripts/bash-solutions, that can find duplicates and run under nix:
rmlint-gui package with GUI (may be launched by rmlint --gui or from desktop launcher named Shredder Duplicate Finder)FSlint has a GUI and some other features. The explanation of the duplicate checking algorithm from their FAQ:
1. exclude files with unique lengths
2. handle files that are hardlinked to each other
3. exclude files with unique md5(first_4k(file))
4. exclude files with unique md5(whole file)
5. exclude files with unique sha1(whole file) (in case of md5 collisions).
sudo apt-get install fslint, the installation currently does put fslint-gui into the path and so I can run it from anywhere by just typing fslint-gui. You can find where fslint-gui lives by typing which fslint-gui (it looks like a Python script).
Nov 25, 2014 at 19:29
If your deduplication task is music related, first run the picard application to correctly identify and tag your music (so that you find duplicate .mp3/.ogg files even if their names are incorrect). Note that picard is also available as an Ubuntu package.
That done, based on the musicip_puid tag you can easily find all your duplicate songs.
fdupes -r -d -N to find and delete duplicates. But it's still not identifying many duplicates. How exactly did you do this with the musicip_puid?
Another script that does this job is rmdupe. From the author's page:
rmdupe uses standard linux commands to search within specified folders for duplicate files, regardless of filename or extension. Before duplicate candidates are removed they are compared byte-for-byte. rmdupe can also check duplicates against one or more reference folders, can trash files instead of removing them, allows for a custom removal command, and can limit its search to files of specified size. rmdupe includes a simulation mode which reports what will be done for a given command without actually removing any files.
I use komparator - sudo apt-get install komparator (Ubuntu 10.04+ ) - as GUI-tool for finding duplicates in manual mode.
For Music related duplicate identification and deletion Picard and Jaikoz by http://musicbrainz.org/ is the best solution. Jaikoz I believe automatically tags your music based on the data of the song file. You don't even need the name of the song for it to identify the song and assign all meta data to it. Although the free version can tag only a limited number of songs in one run, but you can run it as many times as you want.
Have you tried
finddup
or
finddup -l
I guess it works fine.
dupeGuru has a dedicated mode for music. It is a cross-platform GUI program and, as of today (February 2021), it is in active development, although it is unclear which releases work on which systems. Check its documentation.
