3

I have two files containing billions of DNA sequence names, the second is a proper subset of the first:

for example,

1°:john mike anna paul laura .... 

2°:john mike paul ...

all the names occupy a single row.

My question is, how can I get the names contained in the first file that are not in the second one?

thank u all!

3
  • This is not an Ubuntu specific answer. Anyway, you should look at importing those two files into a spreadsheet (you can use Libreoffice Calc, for example). If the names are seperated by any given symbol, you can seperate them into columns with Find/Replace (Symbol -> \n) The first file imported to the first column and the second to the second column. You can compare the content from side-by-side cells. O the third column insert this function =If(A1=B1;1;0) and copy it until the end. This function will will give a negative result if both cells are not equal. Then use Find to look for a 0. Sep 19, 2014 at 13:26
  • 3
    @DavidM.Sousa importing billions of names into a spreadsheet when you already have them in such an easily parseable format is like shooting yourself in the foot.
    – terdon
    Sep 19, 2014 at 13:38
  • You are right. Was just trying to help :) Sep 19, 2014 at 17:25

4 Answers 4

2

This is very simple to do but your life will be much easier if you have one name per line instead of a space separated list. There are many excellent utilities for manipulating text files in Linux, it is one of the things that all *nixes excel at, but most expect one item per line. So, most of my solutions will start with modifying the files accordingly.

Change your files to have one name per line:

sed 's/ /\n/g' file > newfile

or, to modify the original file

sed -i 's/ /\n/g' file

Once you've done that, any of these will give you what you want:

  1. grep

    $ grep -vFwf file2 file1
    anna
    laura
    
  2. comm or diff

    $ comm -23 <(sort file1) <(sort file2)
    anna
    laura
    
    
    $ diff file1 file2 | grep -Po '<\s+\K.*'
    anna
    laura
    
  3. awk

    $ awk '(NR==FNR){a[$1]++; next}!($1 in a){print}' file2 file1 
    laura
    anna
    
  4. Perl

    $ perl -lne 'BEGIN{open(A,"file2"); while(<A>){chomp; $k{$_}++}} print unless $k{$_}' file2 file1
    laura
    anna
    

    or

    $ perl -lne '$k{$_}++; END{map{print unless $k{$_}>1}keys(%k)}' file2 file1
    laura
    anna
    

If you really don't want to change the format of your file (but really, you should), you can do something like

awk '{for (i=1;i<=NF;i++){a[$i]++}}END{for(n in a){if(a[n]<2){print n}}}' file2 file1

or

perl -lane '$k{$_}++ for @F; END{map{print if $k{$_}<2} keys(%k)}' file1 file2
2
  • Boyo are Perl and Awk solutions are complicated to look at! :) Sep 19, 2014 at 14:35
  • @RPiAwesomeness yeah, in this particular case :)
    – terdon
    Sep 19, 2014 at 14:54
2

If they're sorted and separated by newlines, you could use comm show you the lines that are unique to file1:

comm -23 file1 file2

A demonstration:

$ comm -23 <(echo -e 'john\nmike\nanna\npaul\nlaura'|sort) <(echo -e 'john\nmike\npaul'|sort)
anna
laura

Or you could diff to do much the same thing (the grep is looking for line deletions):

diff sorted-file-1 sorted-file-2 | grep -oP '(?<=< ).+'

If you need to avoid sorting or you're dealing with serious numbers, I'd turn to a proper language to do dictionary-based lookups. Simple python example:

file2 = {}
with open("file2") as f:
    for line in f:
        file2[line] = 0

with open("file1") as f:
    for line in f:
        if not line in file2:
            print line

Anything bigger than that and you might want to look at an actual database and some simple SQL. They're geared for big data.

2

And the python option: regardless if all words are in one line or on separate lines:

#!/usr/bin/env python3

import sys

f1 = sys.argv[1]; f2 = sys.argv[2]

def read(f):
    with open(f) as content:
        return content.read().split()

for item in [w for w in read(f1) if not w in read(f2)]:
    print(item)

Copy the script into an empty file, save it as showdiff.py make it executable and run it by the command:

/path/to/showdiff.py file1 file2

anna
laura

Note

Not the question, but too much connected to leave out:

If you need to list the differences mutually, (not only words in file1 that do not appear in file2, but also words in file2 that do not appear in file1), The script below should be used:

#!/usr/bin/env python3

import sys

f1 = sys.argv[1]; f2 = sys.argv[2]

def read(f):
    with open(f) as content:
        return content.read().split()

wds1 = read(f1); wds2 = read(f2); allwords = wds1+wds2
for item in [w for w in allwords if (w in wds1, w in wds2).count(False) == 1]:
    print(item)
0

If you go the python option as suggested by Jacob Vlijm, it is worth using the 'set' (for further info, see https://docs.python.org/3/library/stdtypes.html#set-types-set-frozenset). Basically, once you have created the two sets, you can achieve set math (union, intersection, difference,...)
In this case, set difference is exactly what you require: a new set with all elements that are in one set and not in the other.
The code from Jacob would then be:

#!/usr/bin/env python3

import sys

f1 = sys.argv[1]; f2 = sys.argv[2]

def read_set(f):
    with open(f) as content:
        return set(content.read().split())

for item in read_set(f1) - read_set(f2)]:
    print(item)

Of course, for billions of records, it will take a while... `

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .