I had to consolidate several music directories, and have ended up with many duplicate files. In instances where one is a .flac, I'd like to get rid of the dupes and keep the .flac. I'd like to only compare files that are in the same folder, as these represent albums, which is a useful way for me to organize my music.

I'm running Ubuntu 15.10, with a bash shell.

I would appreciate your help with command line tools. Thanks so much,


Before I answer your question, I have some assumptions: Say you have a directory like this

$ ls -1 myMusics

In the directory above, only music1.flac and music4.flac duplicates will be removed, because music2.mp3, music2b.mp3, music3.flac and music5b.flac have no duplicates to be removed, and although music6 has duplicates, non of them have .flac extension. I've also assumed that there is no file such as music1.backup.flac in your directory. If such files exist, they will be treated as music1.flac duplicates ( if any ).

Based on the assumptions, you can do so:

$ shopt -s extglob
$ for i in `ls | egrep .flac$ | sed 's/\.flac$//'`; do rm "$i".!(flac);done 2>/dev/null
$ shopt -u extglob

Now let's see what the above commands do:

  1. Second one lists all the files that have .flac extension at the end (ls | egrep .flac$),
  2. then removes the .flac from the end of the name (| sed 's/\.flac$//),
  3. then for each of the names that are result of two previous steps, removes all of the files that has that name plus an extension,
  4. except for the ones which has .flac extension. rm "$i".!(flac) removes every file that starts with "$i" and excludes the ones that have .flac extension. If you don't run shopt -s extglob ( that is, if you don't enable extglob ), patterns like "$i"!(.flac) will not have any special meaning and bash searches for the file name "$i"!(.flac) exactly. ( the name "$i" followed by ! and then a ( and so on. )
  5. At the end, you can disable extglob using shopt -u extglob.

You can see the following about extglob in man bash:

If the extglob shell option is enabled using the shopt builtin, several extended pattern matching operators are recognized. In the following description, a pattern-list is a list of one or more patterns separated by a |. Composite patterns may be formed using one or more of the following sub-patterns:

  • ?(pattern-list)
    Matches zero or one occurrence of the given patterns
  • *(pattern-list)
    Matches zero or more occurrences of the given patterns
  • +(pattern-list)
    Matches one or more occurrences of the given patterns
  • @(pattern-list)
    Matches one of the given patterns
  • !(pattern-list)
    Matches anything except one of the given patterns

Also, take a look at this SO question:
How can I use inverse or negative wildcards when pattern matching in a unix/linux shell?

  • Thanks so much! I'm going to study this so I understand what you are teaching me. – mxr Jan 8 '16 at 1:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.