bash file comparison line by line with selected string

Question

let us say i have two files a.txt, b.txt

a.txt

87621   3bde   NDF
87621   2dfg   NDF
87621   cdef   NDF
87621   abcd   NDF

b.txt

93291   abcd   NDF
93291   2dfg   NDF
93291   adbf   NDF
93291   gdrg   NDF

My script should create a file , by matching column 2 of both the files and create a file with matching string in new file c.txt like below

c.txt

2dfg
abcd

I tried several options, but not succeded, can you help !

Answer 1

This reads all the lines in a.txt and keeps track of all of its column 2 values. It then reads b.txt and prints out any repeated values of column 2:

$ awk 'NR==FNR {seen[$2]=1; next} seen[$2]==1 {seen[$2]++; print $2}' a.txt b.txt
abcd
2dfg

The awk command can be divided into the two parts. The first part is:

NR==FNR {seen[$2]=1; next}

awk first tests whether the total number of records (lines) processed, NR, is the same as the number of records processed in this file, FNR. This is true only for the first file processed which is a.txt in this case. Thus, for every line in a.txt, the associative array seen with a key set to the value of the second column, $2, is set to a value of 1. This will be used to indicate that this value of column-2 was "seen" in a.txt. The following command is next which tells awk to skip the rest of the commands and go get the next record.

The second part is:

seen[$2]==1 {seen[$2]++; print $2}

Because of the next command in the first part above, this part is only executed by file b.txt. This part begins with a test: it checks to see if the column-2 value, $2, for this record has been seen before. If is has been seen before once, then the commands in braces are executed. The first command in braces, seen[$2]++ increments the value of seen so that we will never process this value of column-2 again. (Consequently, repeat values of column-2 are ignored.) The second command simply prints the value of column-2 that had been seen before in a.txt.

If we can trust that there are no repeated values for column 2, a simpler script is possible:

$ awk '{print $2}' a.txt b.txt | sort | uniq -d
2dfg
abcd

Answer 2

comm -12 <(awk '{print $2}' a.txt | sort) <(awk '{print $2}' b.txt | sort) > c.txt

Explanation

• <(...) is a process substitution. i.e. the output of the commands within the (...) is substituted and used as the two inputs to comm.
• awk '{print $2}' a.txt prints only the second field of each line.
• sort sorts the output ready for input into comm.
• comm compares two sorted files. The -12 flags suppress the lines that are unique to each file, only printing the common lines (see man comm for more details).

Answer 3

The following perl one-liner will do the job:

perl -lane 'BEGIN{%h}$h{$ARGV."\n".$F[1]}++;END{foreach(keys(%h)){s/.*\n//;$f{$_}++};for(keys(%f)){print if$f{$_}>1}}' a.txt b.txt > c.txt