What is the most efficient way to get the intersection between two files , there is :

sort file1 file2 | uniq -d
  • This doesn't work for file1: "A\nA" and file2 empty. – S Prasanth May 26 '14 at 4:19
  • It's unclear. Please be specific. – Avinash Raj May 26 '14 at 7:41

Use grep command line as following:

grep -Fx -f file1 file2
  • This can give different results if file2 and file1 are interchanged. Example: file1 = "A", file2="A\nA". The number of times a line occurs in both files isn't handled properly. Also, grep should be less efficient than sort since in this case every line in one file will be compared with every line in the other file. – S Prasanth May 26 '14 at 7:04

If you want to combine the contents of two files then run the below command,

awk 'NR==FNR' file1 file2
  • 1
    This just prints the contents of first file. And the question isn't about combining the two files either. – S Prasanth May 26 '14 at 7:38
  • What do you understand from this question? – Avinash Raj May 26 '14 at 7:39
  • 1
    Intersection: Lines which occur in both files, not just one of them. – S Prasanth May 26 '14 at 7:41

Let file1 = 'A\nA' and file2 = 'A\nA\nA'. Is the intersection just 'A' or 'A\nA'? In other words, is the number of times the same line is appears in both the files relevant?

If it is just 'A' this will work:

sort <(sort file1 | uniq) <(sort file2 | uniq) | uniq -d

The problem with the approach given in the question is that even if a line doesn't exist in both files, if it is repeated in one it will appear in the result. So, we need to remove repetitions within files first using sort fileX | uniq.

If it is 'A\nA' then this will work:

sort <(sort file1 | awk '{if ($0 == prev) c++; else c=0; print c, $0; prev = $0;}') <(sort file2 | awk '{if ($0 == prev) c++; else c=0; print c, $0; prev = $0;}') | uniq -d | cut -d' ' -f2-

I can think of a few more one liners (with awk, sort, uniq and cut) but none of them is much simpler than this. This, of course, doesn't mean there aren't any simpler solutions.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.