4

I am trying to extract uncommon text from the file and I have tried this:

awk 'FNR==NR {a[$0]++; next} !a[$0]' 1.txt 2.txt
http://PQR.com
http://example.com

Here are the input files:

File: 1.txt
http://google.com
http://GOOGLE.com
http://example1.com
http://seperate.com
http://pqr.com
File: 2.txt
http://PQR.com
http://example.com
http://google.com

As you can see, http://pqr.com is available in file 1.txt and http://PQR.com in 2.txt. and the result display http://PQR.com, which is common in both file. Hence, how I can show only the uncommon text (not depending upon the cases of the text)?

  • text, meaning lines? if so, how big are the files? – Jacob Vlijm Feb 6 '17 at 8:35
  • @JacobVlijm Yes text refer to a single line of a file. File size say 1.txt is 2 GB and 2.txt is 5GB – Jaffer Wilson Feb 6 '17 at 8:37
  • By "uncommon", do you mean the union of the set differences "(A ∖ B) ∪ (B ∖ A)"? – David Foerster Feb 6 '17 at 11:14
  • @DavidFoerster No it is like only A\B or B\A. No union of two. – Jaffer Wilson Feb 6 '17 at 11:24
7

Since you're already using awk, use tolower to lowercase the lines:

awk 'FNR==NR {a[tolower($0)]++; next} !a[tolower($0)]' foo bar

However, this only prints lines in bar which were not in foo.

Compare with:

$ sort -f bar foo | uniq -iu
http://example.com
http://example1.com
http://seperate.com

With awk, you also need to print every line which was seen only once:

$ awk '{a[tolower($0)]++} END {for (i in a) if (a[i] == 1) print i}' foo bar
http://seperate.com
http://example.com
http://example1.com
| improve this answer | |
  • Thank you muru. Even though the command is slow. It worked. – Jaffer Wilson Feb 6 '17 at 9:27
  • Can I use sort command to get uncommmon values from a single file? As I see the sort command you have mentioned gives uncommon from both the files, which good one. but can I get from one file only as awk works? – Jaffer Wilson Feb 6 '17 at 9:37
  • sort -f foo | uniq -u? – muru Feb 6 '17 at 9:38
  • no not like that. The values which are present in foo and bar are not what I want. I want simply foo - bar. – Jaffer Wilson Feb 6 '17 at 9:45
  • 1
    @JafferWilson then use your original command with tolower, but reverse the order of files. – muru Feb 6 '17 at 9:48
4

This is a pretty simple job for grep:

grep -viFf file2.txt file1.txt
  • -v shows the lines that do not match
  • -i enables case-insensitivity
  • -F makes patterns literal
  • -f file1.txt reads patterns to match from file1.txt, line by line

Example:

% cat file1.txt 
http://google.com
http://GOOGLE.com
http://example1.com
http://seperate.com
http://pqr.com

% cat file2.txt 
http://PQR.com
http://example.com
http://google.com

% grep -viFf file2.txt file1.txt
http://example1.com
http://seperate.com
| improve this answer | |
  • Your command showed aims@aims:~/Downloads/Uncommon1$ grep -viFf 2.txt 1.txt > output.txt grep: memory exhausted – Jaffer Wilson Feb 6 '17 at 9:26
  • Look at the file size, grep is choking in file sizes like this. – Jacob Vlijm Feb 6 '17 at 9:49
4

Here's a Python solution using an efficient set implementation for nice O(n+m) performance (with n and m being the sizes of the two input files).

Code

#!/usr/bin/python3
import sys

with open(sys.argv[1]) as A_file:
    A = frozenset(map(str.casefold, map(str.rstrip, A_file)))

with open(sys.argv[2]) as B_file:
    B = map(str.rstrip, B_file))
    B_minus_A = filter(lambda s: s.casefold() not in A, B)
    print(*B_minus_A, sep='\n')

Usage example

python3 casefold-difference.py 1.txt 2.txt

Explanation

  • The program uses casefold for string comparison yet returns the lines as they appear in 2.txt. Case folding is the recommended way to do case-insensitive comparison of natural language. If that's not what you want (since URLs aren't actually natural language) you may replace it with lower.

  • If 2.txt has a huge amount of lines (that don't appear in 1.txt) the creation of the variadic argument list in the last statement may consume quite a lot of memory and you may be better off to replace it with a loop like this:

    for item in B_minus_A:
        print(item)
    
| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.