Can uniq be used to delete one duplicate line while retaining the other?

Question

I have a file of numbers on different lines. There are many duplications. I would like to delete one line of the duplications while keeping the other line.

uniq -d deletes both lines. Is there a way to only delete one line while leaving the second identical line?

Answer 1

• Sort the numbers with sort
• Remove the duplicates with uniq. You need no option.

Example with digits and two blank lines:

$ cat unsorted

1
2
3
1
2
3
4
3
2
1

$ sort unsorted


1
1
1
2
2
2
3
3
3
4
$ sort unsorted | uniq

1
2
3
4

Answer 2

If you want to output only the first occurrence of each line without sorting, use awk or perl to maintain a hash (associative array)

awk '!a[$0]++' file

or

perl -ne 'print unless $h{$_}++' file

If you want to retain the last occurrence, then do the same thing but read the file backwards and reverse it after:

tac file | awk '!a[$0]++' | tac

For more complicated deletions (such as deleting all but the second of multiple duplicates, while printing all non-duplicate lines) I think you will need to process the file twice.