0

I'm trying to find a way to find and print only lines from a file that don't have duplicates. If this is my file:

A
A
B
B
C
C
Y
Z

I am trying to print out only

Y
Z

Unfortunately, I keep getting

A
B
C
Y
Z

I have tried sort -u, sort | uniq -u, and grep | sort | uniq -u with the same results. I was eventually able to achieve my goal of finding the unique line using uniq -c and looking for the line that only appears one time, but I would like to know how to do this properly in the future.

3
  • Sorry, there should be a line return after each letter in the examples, but I didn't format my post properly May 16, 2017 at 3:22
  • 1
    Fixed that for you
    – wjandrea
    May 16, 2017 at 3:26
  • 4
    sort | uniq -u works fine for me.
    – muru
    May 16, 2017 at 3:27

2 Answers 2

2

AWK solution

$ awk '{arr[$0]++};END{for(var in arr) if (arr[var] == 1) print var}' input.txt                                          
Y
Z
  • {arr[$0]++}; creates associative array of line-number pairs. If a line is unique in the file, array item that corresponds to the line value will be 1, otherwise - greater than 1
  • END block is executed when we have reached end of file. We iterate over array items using for(value in array) loop and print the value if the corresponding array item equals to 1, as mentioned before.

Python 3

Same idea as the awk one. Here we use OrderedDict class to create a dictionary of lines and their counts with preserved order.

#!/usr/bin/env python3
import sys
from collections import OrderedDict

if len(sys.argv) != 2:
   sys.stderr.write(">>> Script requires a file argument")
   sys.exit(1)

for arg in sys.argv[1:]:
    lines = OrderedDict()
    with open(sys.argv[1]) as fd:
        for line in fd:
            tmp = line.strip()
            if tmp in lines.keys():
                lines[tmp] = lines[tmp] + 1
            else:
                lines[tmp] = 1

    for line,count in lines.items():
        if count == 1:
            print(line)

And here it is in action:

$ ./get_unique_lines.py  input.txt                                                                                       
Y
Z

Perl

Again, same idea as Python script, and we're using ordered hash (see also the Tie::IxHash documentation ).

#!/usr/bin/perl
use strict;
use warnings;
use Tie::IxHash;

tie my %linehash, "Tie::IxHash" or die $!;

open(my $fp,'<',$ARGV[0])  or die $!;
while(my $line = <$fp> ){
    chomp $line;
    $linehash{$line}++;
}
close($fp);

for my $key (keys %linehash) {
    printf("%s\n",$key) unless $linehash{$key} > 1;
}

Test run:

$ ./get_unique_lines.pl input.txt                                                                                        
Y
Z

sort and uniq variations

Have been mentioned in the comments multiple times already.

$ sort input.txt | uniq -u                                                                                               
Y
Z

or

$ uniq -u input.txt                                                                                                      
Y
Z
2
  • I appreciate how thorough your answer is. I am looking for an approach that doesn't require scripting, which you were kind enough to provide in addition to the others. It appears that I have been using my uniq command properly and this may just be an issue on the server I'm dealing with. It's just frustrating me because I feel like it should be working, but I'm not getting the proper results. Thank you. May 16, 2017 at 6:02
  • @ChristianShipley Well, unless the server is non-Ubuntu , the uniq should indeed be working. The thing is that different OS have different behavior of the tools even if they're named the same, and if indeed you have different Linux distro asking this on Ask Ubuntu is not very good idea. Regardless of that, in case of Ubuntu and FreeBSD, at least, the -u flag should behave the same. But yes, try contacting the server administrator, maybe they'll figure this one out. Let me know if you have any other questions May 16, 2017 at 6:15
1

Please take a look at this post on Stack Overflow

The answer is

uniq -u test.txt
3
  • This is the reason I've posted this question actually. I've tried this command several times, and I continue to receive A, B, C, Y, and Z in response. I had begun to think I was using it wrong, but it appears I may have to contact the server I'm working on because there seems to be a consensus that my approach should have worked. Thank you May 16, 2017 at 5:45
  • @ChristianShipley uniq -u alone won't be enough, but sort | uniq -u should be fine.
    – muru
    May 16, 2017 at 5:59
  • sort test.txt | uniq -u worked for me. follow the link posted by @user3051574 Oct 20, 2020 at 21:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.