So I'm checking the md5 hash of my files with this as my output:

657cf4512a77bf47c39a0482be8e41e0  ./dupes2.txt
657cf4512a77bf47c39a0482be8e41e0  ./dupes.txt
8d60a927ce0f411ec94ac26a4785f749  ./derpina.txt
15f63928b8a1d5337137c38b5d66eed3  ./foo.txt
8d60a927ce0f411ec94ac26a4785f749  ./derp.txt

However, after running find . -type f -exec md5sum '{}' ';' | uniq -w 33 to find the unique hashes I get this:

657cf4512a77bf47c39a0482be8e41e0  ./dupes2.txt
8d60a927ce0f411ec94ac26a4785f749  ./derpina.txt
15f63928b8a1d5337137c38b5d66eed3  ./foo.txt
8d60a927ce0f411ec94ac26a4785f749  ./derp.txt

From my understanding, only one of either derpina.txt or derp.txt should be showing up since their hashes are the same. Am I missing something? Can anyone enlighten me as to why it outputs like this?


4 Answers 4


You need to use sort before uniq:

find . -type f -exec md5sum {} ';' | sort | uniq -w 33

uniq only removes repeated lines. It does not re-order the lines looking for repeats. sort does that part.

This is documented in man uniq:

Note: uniq does not detect repeated lines unless they are adjacent. You may want to sort the input first, or use sort -u without uniq.

  • 1
    uniq should be aliased as sort -u by default in all systems. If at all it always needs "sort" for it to work properly.
    – Devaroop
    Apr 12, 2019 at 10:19
  • That change would lessen some confusion. On the other hand, uniq has many features not available with sort -u. Also, there are cases where one wants to use uniq without sort.
    – John1024
    Apr 12, 2019 at 19:59

The input for uniq needs to be sorted. So for the example case,

find . -type f -exec md5sum '{}' ';' | sort | uniq -w 33

would work. The -w (--check-chars=N) makes the lines unique only regarding the first column; This option works for this case. but the possibilities to specify the relevant parts of the line for uniq are limited. For example, there are no options to specify working on some column 3 and 5, ignoring column 4.

The command sort has an option for unique output lines itself, and the lines are unique regarding the keys used for sorting. This means we can make use of the powerful key syntax of sort to define regarding which part the lines should be uniq.

For the example,

find . -type f -exec md5sum '{}' ';' | sort -k 1,1 -u

gives just the same result, but the sort part is more flexible for other uses.


The uniq command won't work unless input is buffered (e.g. using sort) or duplicated lines are next to each other. So find below few solutions.


Here is syntax for awk:

command | awk '!a[$0]++{print}'

For repeated use, you can define the following shell alias:

alias unicat='awk "!a[\$0]++{print}"'

then run: command | unicat.

Parallel syntax (for quicker parsing):

command | time parallel --block 100M --pipe awk '!a[\$0]++{print}'

Test command (100 lines filtered out of 200):

echo | pee "seq 1 100" "seq 1 100" | awk '!a[$0]++{print}' | wc -l


Try the following Perl script:

command | perl -e 'while(<>){if(!$s{$_}){print $_;$|=1;$s{$_}=1;}}'

Shell alias syntax for repeated use:

alias unicat='perl -e '\''while(<>){if(!$s{$_}){print $_;$|=1;$s{$_}=1;}}'\'''

then run: command | unicat.


For alternative to uniq, you can also use karrick's unique written in Go (it's using similar approach as above, but it's much faster and uses less memory than Perl/AWK solutions):


command | unique

Cons: When processing huge data, it's using a lot of memory, as it's hashing each item and adding it into array.

Workaround can be to have short lived instances per certain block size (if few duplicates are acceptable), e.g.

command | parallel --block 100M -j 1 --pipe unique

or using split (10M per chunk), e.g.

command | split -b 10M --filter="unique"


Use quniq utility to remove duplicated. It works better than uniq, as it buffers the input lines.

Cons: It seems to be stuck till end of the pipe.


If you're looking for faster solution to sort then remove duplicates, use huniq.

More similar projects:


Or you could install killdupes, my program to destroy every last effing duplicate there is!



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.