0

I have a file of numbers on different lines. There are many duplications. I would like to delete one line of the duplications while keeping the other line.

uniq -d deletes both lines. Is there a way to only delete one line while leaving the second identical line?

1
  • Shell commands use a different way of thinking: Input -> Process -> Output. Read File1, perform uniq, write the output as File2.
    – user535733
    Jul 7, 2018 at 21:49

2 Answers 2

1
  • Sort the numbers with sort
  • Remove the duplicates with uniq. You need no option.

Example with digits and two blank lines:

$ cat unsorted

1
2
3
1
2
3
4
3
2
1

$ sort unsorted


1
1
1
2
2
2
3
3
3
4
$ sort unsorted | uniq

1
2
3
4
1
  • 2
    or with GNU, just sort -u unsorted to "output only the first of an equal run" Jul 7, 2018 at 22:18
1

If you want to output only the first occurrence of each line without sorting, use awk or perl to maintain a hash (associative array)

awk '!a[$0]++' file

or

perl -ne 'print unless $h{$_}++' file

If you want to retain the last occurrence, then do the same thing but read the file backwards and reverse it after:

tac file | awk '!a[$0]++' | tac

For more complicated deletions (such as deleting all but the second of multiple duplicates, while printing all non-duplicate lines) I think you will need to process the file twice.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.