uniq command not working properly?

Question

So I'm checking the md5 hash of my files with this as my output:

657cf4512a77bf47c39a0482be8e41e0  ./dupes2.txt
657cf4512a77bf47c39a0482be8e41e0  ./dupes.txt
8d60a927ce0f411ec94ac26a4785f749  ./derpina.txt
15f63928b8a1d5337137c38b5d66eed3  ./foo.txt
8d60a927ce0f411ec94ac26a4785f749  ./derp.txt

However, after running find . -type f -exec md5sum '{}' ';' | uniq -w 33 to find the unique hashes I get this:

657cf4512a77bf47c39a0482be8e41e0  ./dupes2.txt
8d60a927ce0f411ec94ac26a4785f749  ./derpina.txt
15f63928b8a1d5337137c38b5d66eed3  ./foo.txt
8d60a927ce0f411ec94ac26a4785f749  ./derp.txt

From my understanding, only one of either derpina.txt or derp.txt should be showing up since their hashes are the same. Am I missing something? Can anyone enlighten me as to why it outputs like this?

Answer 1

You need to use sort before uniq:

find . -type f -exec md5sum {} ';' | sort | uniq -w 33

uniq only removes repeated lines. It does not re-order the lines looking for repeats. sort does that part.

This is documented in man uniq:

Note: uniq does not detect repeated lines unless they are adjacent. You may want to sort the input first, or use sort -u without uniq.

Answer 2

The input for uniq needs to be sorted. So for the example case,

find . -type f -exec md5sum '{}' ';' | sort | uniq -w 33

would work. The -w (--check-chars=N) makes the lines unique only regarding the first column; This option works for this case. but the possibilities to specify the relevant parts of the line for uniq are limited. For example, there are no options to specify working on some column 3 and 5, ignoring column 4.

The command sort has an option for unique output lines itself, and the lines are unique regarding the keys used for sorting. This means we can make use of the powerful key syntax of sort to define regarding which part the lines should be uniq.

For the example,

find . -type f -exec md5sum '{}' ';' | sort -k 1,1 -u

gives just the same result, but the sort part is more flexible for other uses.

Answer 3

The uniq command won't work unless input is buffered (e.g. using sort) or duplicated lines are next to each other. So find below few solutions.

AWK

Here is syntax for awk:

command | awk '!a[$0]++{print}'

For repeated use, you can define the following shell alias:

alias unicat='awk "!a[\$0]++{print}"'

then run: command | unicat.

Parallel syntax (for quicker parsing):

command | time parallel --block 100M --pipe awk '!a[\$0]++{print}'

Test command (100 lines filtered out of 200):

echo | pee "seq 1 100" "seq 1 100" | awk '!a[$0]++{print}' | wc -l

Perl

Try the following Perl script:

command | perl -e 'while(<>){if(!$s{$_}){print $_;$|=1;$s{$_}=1;}}'

Shell alias syntax for repeated use:

alias unicat='perl -e '\''while(<>){if(!$s{$_}){print $_;$|=1;$s{$_}=1;}}'\'''

then run: command | unicat.

unique

For alternative to uniq, you can also use karrick's unique written in Go (it's using similar approach as above, but it's much faster and uses less memory than Perl/AWK solutions):

• https://github.com/karrick/unique

Usage:

command | unique

Cons: When processing huge data, it's using a lot of memory, as it's hashing each item and adding it into array.

Workaround can be to have short lived instances per certain block size (if few duplicates are acceptable), e.g.

command | parallel --block 100M -j 1 --pipe unique

or using split (10M per chunk), e.g.

command | split -b 10M --filter="unique"

quniq

Use quniq utility to remove duplicated. It works better than uniq, as it buffers the input lines.

• Project page: https://github.com/syucream/quniq
• Install by: go install github.com/syucream/quniq@latest

Cons: It seems to be stuck till end of the pipe.

huniq

If you're looking for faster solution to sort then remove duplicates, use huniq.

• Project page: https://github.com/koraa/huniq
• Install by: cargo install huniq

---

More similar projects:

• https://github.com/mitsutoshi/uni (written in Rust)
• https://github.com/whitfin/runiq (written in Rust)
• https://github.com/miolini/uniqbloom (written in Go, using Bloom filter)

Answer 4

Or you could install killdupes, my program to destroy every last effing duplicate there is!

https://github.com/batchmcnulty/killdupes

:-)