1

I have two files - file1 & file2.

file1 contains (only words):

ABC 
YUI 
GHJ 
I8O

file2 contains many paragraphs:

dfghjo ABC kll 
njjgg bla bla GHJ 
njhjckhv chasjvackvh .. 
ihbjhi hbhibb jh jbiibi 

I am using the command below to get the matching lines which contains word from file1 in file2:

grep -Ff file1 file2 

(Gives output of lines where words of file1 found in file2)

I also need the words from file1 which are not found in file2.

Can anyone help in getting this output:

YUI 
I8O

I am looking for a one liner command (via grep, awk, sed), as I am using pssh command and can't use while or for loops.

1

Here's one way in awk:

$ awk 'NR==FNR{a[$1]++; next}{for(i in a){if($0 ~ i){found[i]++}}}END{for(i in a){if(!found[i]){print i}}}' file1 file2 
YUI
I8O

Or, a bit more legibly:

$ awk 'NR==FNR{
        a[$1]++; 
        next
       }
       {
        for(i in a){
            if($0 ~ i){
                found[i]++
            }
        }
       }
       END{
        for(i in a){
            if(!found[i]){
                print i
            }
        }
       }' file1 file2 
YUI
I8O

Explanation

  • NR==FNR : NR is the current line number and FNR is the current line number of the current file. When processing multiple files, the two will be equal only while reading the first file. So this is an easy way of saying "do this for the 1st file only".
  • a[$1]++; next : while reading the first file, save each word (the first and only field) in the array a and then skip to the next line. The next also ensures that the rest of the command is not run for the first file.
  • for(i in a){ if($0 ~ i){ found[i]++ } }: For each of the words found in the first file (the keys of array a), check if the current line matches that word. If it does, save the word in the found array. This is run for each line of the second input file.
  • END{ }: do this after you've processed all input files.
  • for(i in a){ if(!found[i]){ print i } }: for each of the words in a, if the word is not also in the found array, print that word.

Alternatively, you can use some of the core Linux utilities:

$ grep -hoP '\w+' file1 file2 | sort | uniq -u | xargs -I{} grep {} file1
I8O
YUI 

Explanation

$ grep -hoP '\w+' file1 file2
ABC
YUI
GHJ
I8O
dfghjo
ABC
kll
njjgg
bla
bla
GHJ
njhjckhv
chasjvackvh
ihbjhi
hbhibb
jh
jbiibi

This will print all the words found in each file. The -o flag means "only print the matching portion of the line", the -P enables Perl Compatible Regular Expressions (PCRE) which let us use \w to mean "any word character" (so letters, numbers, _).

$ grep -hoP '\w+' file1 file2 | sort | uniq -u
chasjvackvh
dfghjo
hbhibb
I8O
ihbjhi
jbiibi
jh
kll
njhjckhv
njjgg
YUI

Now we pass the output of the previous command through sort and uniq -u to keep only unique matches: these are the words that are only present in one of the two files.

$ grep -hoP '\w+' file1 file2 | sort | uniq -u | xargs -I{} grep {} file1
I8O
YUI 

Finally, we feed this list of unique words to xargs and have it grep each of them in file1. Only those unique words that are present in file1 will be returned, and unique words present in file1 are therefore not present in file2.

  • 2
    @Sin15 you're welcome. Have a look at the updated answer, I added a much shorter and simpler version as well. Also, if this answer solved your issue, please take a moment to accept it by clicking on the checkmark on the left. That will mark the question as answered and is the way that thanks are conveyed on the Stack Exchange sites. Feel free to wait for more answers and accept another one, just remember to eventually accept one. – terdon Feb 19 at 9:43
0

try this command:

grep -oFf file1 file2 | grep -vFf - file1

where first use file1 as PATTERN and get only the part of a matching line that matches PATTERN in file2, first command give you:

ABC
GHJ

then use this output as input file PATTERN and search line in file1 that doesn't match PATTERN, and you will get:

YUI
I8O

Tested on Red Hat Enterprise Linux ES release 4 (Nahant Update 3)

0

Using Perl:

Main algorithm + Perl equivalent :

slurp f2                              // $f2 = `cat f2`
for all lines in f1                   // perl -nle  '....' f1
   print if f2 does not have it       //    print  if $f2 !~ /$_/

All together:

perl -nle 'BEGIN {$f2=`cat f2`}  
           print  if $f2 !~ /$_/'  f1

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.