3

I have a file that looks like this

AAA_21               PF13304.1  x_00004
AAA_22               PF13401.1  x_00004
SMC_N                PF02463.14 x_00004
AAA_29               PF13555.1  x_00004
DUF258               PF03193.11 x_00005
AAA_15               PF13175.1  x_00005
AAA_21               PF13304.1  x_00005
AAA_22               PF13401.1  x_00005
SMC_N                PF02463.14 x_00005
AAA_15               PF13175.1  x_00006
AAA_21               PF13304.1  x_00006
AAA_22               PF13401.1  x_00007
SMC_N                PF02463.14 x_00007

Now, for each block of lines that have the same string in column 3 (e.g. x_00004), I want to grep only the lines containing specific strings if they are present together in the block.

So, I know that I can use grep -f <file containing string> <file to scan> But I cannot find a way for applying the first action. I guess awk will help me here, but I do not really know how.

I would like to have something like:

AAA_21               PF13304.1  x_00004
AAA_22               PF13401.1  x_00004
AAA_21               PF13304.1  x_00005
AAA_22               PF13401.1  x_00005

So basically greping the lines containing PF13304.1 or PF13401.1 only if they are sharing field 3.

I use PF13304.1 and PF13401.1 as example, because sometimes I look for the presence of 3 strings in the block. One problem is that the string I am looking for are not always consecutive in the file I want to scan.

All the strings I want to grep are reported in a txt file as well. I can organize them as I want to match the grep command.

Instead the line containing

AAA_21               PF13304.1  x_00006
AAA_22               PF13401.1  x_00007

Should not be included because the strings I want to grep do not share field 3, meaning they are not both present in the subgroups x_00006 or x_00007

So, from the logical point of view I want to

  1. open the file
  2. divide the lines in groups according with field 3, create group that have the same string in field 3
  3. in this subgroups grep the strings I am looking for only if they are all present in each block
  • so you don't want this lines AAA_22 PF13401.1 x_00006 and AAA_22 PF13304.1 x_00005, if they were exist in your file, right? – αғsнιη Feb 16 '15 at 12:22
  • yes I want to have them as well. basically, looking in subset of lines, defined by column 3. In this subset I want to grep the string I am interest in. – efrem Feb 16 '15 at 12:41
  • Now I really don't understand your question. Sorry and I think this doesn't what you want grep '.*PF13\(304\|401\)\.1.*' file in fact, ehm? – αғsнιη Feb 16 '15 at 12:44
  • Can you show an instance where PF13304.1 will not appear in the output? – muru Feb 16 '15 at 12:46
  • 1
    But as I commented in previously and your reply you wants them as well while in body of question you said I don't want that lines? and now it's more unclear at least for me. – αғsнιη Feb 16 '15 at 12:55
1

Can be done in Python fairly easily:

$ cat input.txt | ./find_strings.py PF13304.1 PF13401.1                                                                  
AAA_21               PF13304.1  x_00004
AAA_22               PF13401.1  x_00004
AAA_21               PF13304.1  x_00005
AAA_22               PF13401.1  x_00005
AAA_21               PF13304.1  x_00006
AAA_22               PF13401.1  x_00007

Contents of find_strings.py:

#!/usr/bin/env python
import sys
strings=sys.argv[1:]
for line in sys.stdin:
    for string in strings:
         if string in line:
             print line.strip()

The way this words is that we redirect contents of input file to script's stdin stream, read the stream line by line, and for each line we make a lookup in the list of arguments that we provide on command-line. Fairly simple approach

0

Certainly not as simple as grep. This program:

  • scans the text file, accumulating "blocks" where the 3rd field are the same string
  • when it has found a block, invoke grep and collect the output
  • if the number of lines in the output is the same as the number of search terms, output grep's output
awk '
  function grep(block,    m, grep_out, cmd, line, i) {
    m = 0
    delete grep_out

    cmd = "grep -f " ARGV[1]    # define the grep command
    print block |& cmd          # invoke grep, and send the block of text as stdin
    close(cmd, "to")            # close greps stdin so we can start reading the output

    # read from grep until no more output
    while ((cmd |& getline line) > 0)
      grep_out[m++] = line
    close(cmd)

    # did grep find all search terms?  If yes, print the output 
    if (length(grep_out) == nterms)
      for (i=0; i<m; i++) 
        print grep_out[i]
  }

  # read the search terms file, just to count the number of lines
  NR == FNR {
    nterms++
    next
  }

  # if we detect a new block, call grep and start a new block
  section != $3 {
    if (block) grep(block)
    block = ""
    section = $3
  } 

  {block = block $0 RS}   # accumulate the lines in this block

  END {if (block) grep(block)}       # also call grep at end of file

' fileContainingStrings fileToScan 

produces this output:

AAA_21               PF13304.1  x_00004
AAA_22               PF13401.1  x_00004
AAA_21               PF13304.1  x_00005
AAA_22               PF13401.1  x_00005
0

So if I understand you correctly, you want to find all subgroups that contain ALL patterns you specify. This can be done with sort and awk, e.g.:

# make sure subgroups are adjacent 
sort -k3,3 infile |

# add a newline between subroups, this allows the next 
# invocation of awk to read each subgroup as a record
awk 'NR > 1 && p!=$3 { printf "\n" } { p=$3 } 1' |   

# match the desired patterns and print the subgroup name
awk '/\<PF13304\.1\>/ && /\<PF13401\.1\>/ { print $3 }' RS=

Output:

x_00004
x_00005

Based on the above output you can now extract the relevant lines from infile, e.g. add the following to the above pipe:

while read sgrp; do
  grep -E "\b(PF13304\.1|PF13401\.1)\b +$sgrp\$" infile
done

Output:

AAA_21               PF13304.1  x_00004
AAA_22               PF13401.1  x_00004
AAA_21               PF13304.1  x_00005
AAA_22               PF13401.1  x_00005
0

The following awk script matches literal strings listed one per line in match_file, against data_file

awk 'function endgroup() {
         gmc=0                              # group match count
         for( gi=1; gi<=gz; gi++ ) {        # step through all lines in a group
             split(group[gi],g)             # split one group line 
             for( lix in lms )              # for each literal match string index 
                 if( lix == g[2] )          # does literal match string = group record $2  
                     mrec[++gmc]=group[gi]  # group matched record array, and inc match count
         } 
         if( gmc==lmz ) for( mri=1; mri<=lmz; mri++ ) print mrec[mri]
         delete group; gz=0
     }

     BEGIN{ p3=FS } # an impossible previous value of $3 of "data_file"

     # process "match_file"
     NR==FNR { lms[$0]   # build array with literal match strings as indices
               lmz++     # literal match strings array size 
               next } 
     # process "data_file"
     p3!=$3 && p3!=FS { endgroup() }
     { group[++gz]=$0; p3=$3 }

     END{ if( p3!=FS ) endgroup() }
' match_file data_file

Output:

AAA_21               PF13304.1  x_00004
AAA_22               PF13401.1  x_00004
AAA_21               PF13304.1  x_00005
AAA_22               PF13401.1  x_00005
0

Something like this?

awk '(/x_00004/ || /x_00005/) && (/PF13401.1/ || /PF13304.1/)' your_file

or this, principally the same but with a more readable grouping

awk '(/x_00004/ && (/PF13401.1/ || /PF13304.1/)) || (/x_00005/ && (/PF13401.1/ || /PF13304.1/))' your_file

Example

The input file

cat foo

AAA_21               PF13304.1  x_00004
AAA_22               PF13401.1  x_00004
SMC_N                PF02463.14 x_00004
AAA_29               PF13555.1  x_00004
DUF258               PF03193.11 x_00005
AAA_15               PF13175.1  x_00005
AAA_21               PF13304.1  x_00005
AAA_22               PF13401.1  x_00005
SMC_N                PF02463.14 x_00005
AAA_15               PF13175.1  x_00006
AAA_21               PF13304.1  x_00006
AAA_22               PF13401.1  x_00007
SMC_N                PF02463.14 x_00007

The command

awk '(/x_00004/ || /x_00005/) && (/PF13401.1/ || /PF13304.1/)' foo

AAA_21               PF13304.1  x_00004
AAA_22               PF13401.1  x_00004
AAA_21               PF13304.1  x_00005
AAA_22               PF13401.1  x_00005

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.