0

To illustrate it further we have two file contents:

file1

hello
1_hello 
2_hello
world
1_world
2_world
hello1
1_hello1
2_hello1
world1
1_world1
2_world1

file2

This
hello
1_hello
2_hello
is world
1_world
2_world
my
hello1
1_hello1
2_hello1
word
world1
1_world1
2_world1
file 

what I want is to iterate the first column of file1 and remove the entries which are matching in the file2 and produce the output like:

This
is
my 
word
file

How can I proceed?

5

You want to use awk to read file1 and remember all its words. Then read file2 and output any words that were not seen from file1:

gawk -v RS='[[:space:]]+' 'NR==FNR {words[$1]=1; next} !($1 in words)' file1 file2

That uses any sequence of whitespace as the record separator, so every word is treated as a separate "line". This now GNU awk specific, but that's the default awk on Ubuntu

  • It's just plain awesome, thank you for sharing this knowledge! – dessert Sep 20 '17 at 18:05
  • @dessert: If you want an even more elegant answer, see mine. ;-P – David Foerster Oct 2 '17 at 10:19
3

You can use grep -f FILE to obtain patterns from a file FILE. In your case I recommend some additional flags (see explanation below) for the following final grep invocation:

grep -v -x -F -f file1 -- file2
  • -f FILE – Obtain patterns from FILE, one per line.

  • -F – Interpret the pattern as a list of fixed strings, separated by newlines, any of which is to be matched.

  • -x – Select only those matches that exactly match the whole line. (You may want to remove this option if you want to allow partially matching lines.)

  • -v – Invert the sense of matching, to select non-matching lines.

Source: the GNU grep documentation or alternatively its manual page

All options used above are specified by POSIX and don't require GNU extensions.

  • Elegant indeed! +1 ;) – dessert Oct 2 '17 at 10:53
1

This command line should do the trick:

while read -r word; do sed -e "s/\<$word\>//g" -e '/^\s*$/d' file2 -i; done < file1

After the execution of the above command, the output file - file2 - should look as this:

This
is
my
word
file

More readable version of the above command line:

while read -r word; do \
    sed -e "s/\<$word\>//g" -e '/^\s*$/d' file2 -i; \
done < file1

The while loop reads the file line by line - < file1. The value of each line is used as a value of the temporary created variable, called $word - -r word. This variable is used as an argument within the command sed and it is substituted [s] by a blank value in file2, the first expression: "s/\<$word\>//g" = s/<source_value>/<replacement_value>/g. The g flag means - apply the replacement to all matches. Then if there is a blank line within the file it is removed - the second expression: '/^\s*$/d'.

We need to use the syntax \<...\>to find only the exact matches. We need double quote marks - "..." - for the first expression because $word is a name of variable and we want to expanded it as its value within the sed command.

The option -i means that the changes will be made in their places within the target file - file2. If this option is removed the result will be thrown to stdout but it will be not meaningful. The option -i.bak is not applicable to this scenario, because the target file will be overwritten many times, so you should create a backup copy in advance.

0

Use grep with -v (invert) flag.

With using extended regex (-E), you can make a |-separated list of patterns, that you wish to exclude , e.g. pattern1|pattern2|pattern3:

grep -v -E "^[0-9]|world|hello" file
0

Although the removing criteria is not very clear, I assume that the parts to remove are (www\n1_www\n2_www) for all word www (correct me if I am wrong)

Using (gnu)sed:

sed -zr 's/(\w+)\n1_\1\n2_\1\n//g' ex

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.