0

I have a file of numbers on different lines. There are many duplications. I would like to delete one line of the duplications while keeping the other line.

uniq -d deletes both lines. Is there a way to only delete one line while leaving the second identical line?

1
  • Shell commands use a different way of thinking: Input -> Process -> Output. Read File1, perform uniq, write the output as File2.
    – user535733
    Jul 7, 2018 at 21:49

2 Answers 2

1
  • Sort the numbers with sort
  • Remove the duplicates with uniq. You need no option.

Example with digits and two blank lines:

$ cat unsorted

1
2
3
1
2
3
4
3
2
1

$ sort unsorted


1
1
1
2
2
2
3
3
3
4
$ sort unsorted | uniq

1
2
3
4
1
  • 2
    or with GNU, just sort -u unsorted to "output only the first of an equal run" Jul 7, 2018 at 22:18
1

If you want to output only the first occurrence of each line without sorting, use awk or perl to maintain a hash (associative array)

awk '!a[$0]++' file

or

perl -ne 'print unless $h{$_}++' file

If you want to retain the last occurrence, then do the same thing but read the file backwards and reverse it after:

tac file | awk '!a[$0]++' | tac

For more complicated deletions (such as deleting all but the second of multiple duplicates, while printing all non-duplicate lines) I think you will need to process the file twice.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .