Use a list of words to grep in an other list

Question

I've got a list with 250 lines in it. I have to run all of them through a web server to get a list of output. This list, however returns many more lines, than I'm interested in. Say, my list.txt is:

a.1
b.1
etc

then the output is output.txt:

a.1 a b c
a.2 b a b
a.3 d k o
b.1 b o p
b.2 o i y
b.3 p i y
etc

Is it possible to use the grep command to search for all words in list.txt in the output.txt and then generate "the wanted" list wanted.txt? I need the entire line in my output.txt I'm new in scripting, but what I'd like is something such as

grep list.txt output.txt > wanted.txt

I haven't been able to find any examples of this

Answer 1

I'd ignore grep for this one. It's good for regular expressions but it doesn't look like you really need that here. comm can compare two files and show you intersections. Using your exact examples:

$ comm -12 list.txt output.txt 
a.1
b.1
etc

• -1 suppresses lines unique to the first file
• -2 suppresses lines unique to the second file
• -3 suppresses lines common to both
• To get just the common lines, we suppress the unique lines, hence -12

This is faster than any grep will be but it relies (heavily) on the files being sorted. If they aren't, you can pre-sort them but that will alter the output so it's sorted too.

comm -12 <(sort list.txt) <(sort output.txt) 

---

Alternatively, this answer from iiSeymour will let you do it with grep. The flags ask for an input file and force a fixed-string, full-word search. This won't rely on order but will be based on the output.txt order. Reverse the files if you want them in the order of the list.txt.

$ grep -wFf list.txt output.txt 
a.1
b.1
etc

---

If your list.txt is really big, you might have to tackle this a little more iteratively and pass each line to grep separately. This will massively increase processing time. In the above you'd be reading output.txt once, but this way you'd read and process it for every list.txt line. It's horrible... But it might be your only choice. On the upside, it does then sort things by the list.txt order.

$ while read line; do grep -wF "$line" output.txt; done < list.txt
a.1
b.1
etc