How to extract uncommon case insensitive text from two files?

Question

I am trying to extract uncommon text from the file and I have tried this:

awk 'FNR==NR {a[$0]++; next} !a[$0]' 1.txt 2.txt
http://PQR.com
http://example.com

Here are the input files:

File: 1.txt
http://google.com
http://GOOGLE.com
http://example1.com
http://seperate.com
http://pqr.com
File: 2.txt
http://PQR.com
http://example.com
http://google.com

As you can see, http://pqr.com is available in file 1.txt and http://PQR.com in 2.txt. and the result display http://PQR.com, which is common in both file. Hence, how I can show only the uncommon text (not depending upon the cases of the text)?

Answer 1

Since you're already using awk, use tolower to lowercase the lines:

awk 'FNR==NR {a[tolower($0)]++; next} !a[tolower($0)]' foo bar

However, this only prints lines in bar which were not in foo.

Compare with:

$ sort -f bar foo | uniq -iu
http://example.com
http://example1.com
http://seperate.com

With awk, you also need to print every line which was seen only once:

$ awk '{a[tolower($0)]++} END {for (i in a) if (a[i] == 1) print i}' foo bar
http://seperate.com
http://example.com
http://example1.com

Answer 2

This is a pretty simple job for grep:

grep -viFf file2.txt file1.txt

• -v shows the lines that do not match
• -i enables case-insensitivity
• -F makes patterns literal
• -f file1.txt reads patterns to match from file1.txt, line by line

Example:

% cat file1.txt 
http://google.com
http://GOOGLE.com
http://example1.com
http://seperate.com
http://pqr.com

% cat file2.txt 
http://PQR.com
http://example.com
http://google.com

% grep -viFf file2.txt file1.txt
http://example1.com
http://seperate.com

Answer 3

Here's a Python solution using an efficient set implementation for nice O(n+m) performance (with n and m being the sizes of the two input files).

Code

#!/usr/bin/python3
import sys

with open(sys.argv[1]) as A_file:
    A = frozenset(map(str.casefold, map(str.rstrip, A_file)))

with open(sys.argv[2]) as B_file:
    B = map(str.rstrip, B_file))
    B_minus_A = filter(lambda s: s.casefold() not in A, B)
    print(*B_minus_A, sep='\n')

Usage example

python3 casefold-difference.py 1.txt 2.txt

Explanation

• The program uses casefold for string comparison yet returns the lines as they appear in 2.txt. Case folding is the recommended way to do case-insensitive comparison of natural language. If that's not what you want (since URLs aren't actually natural language) you may replace it with lower.

• If 2.txt has a huge amount of lines (that don't appear in 1.txt) the creation of the variadic argument list in the last statement may consume quite a lot of memory and you may be better off to replace it with a loop like this:

  for item in B_minus_A:
      print(item)