2

is there a way of displaying the lines containing specific strings (like reverent:::N in the example below) that do not occur for N times in a specific file?

I know I can count the occurrence of some string with grep, write it to a new file and then count the lines with wc -l. However, I have a huge file where I need to filter out the lines where the string in column 1 does not appear 5 times in column 1 of the file.

Suggestions are very much appreciated.

edit The lines in file_1.txt are alphabetically sorted by the first column. The file contains about 1.000.000 lines, I could also break it down into a few small files, of course. The order of the output does not matter.

file_1.txt

realism:::N     1.33    depth:::N               3.11    341
realism:::N     1.33    problem:::N             2.68    335
realism:::N     1.33    now:::ADV               1.48    335
realism:::N     1.33    life:::N                2.69    334
renowned:::ADJ  1.41    be:::V                  1.85    15760
renowned:::ADJ  1.41    internationally:::ADV   2.23    9134
renowned:::ADJ  1.41    world:::N               4.36    6736
renowned:::ADJ  1.41    most:::ADV              2.38    5482
reverent:::ADJ  1.5     use:::V                 2.78    25
reverent:::ADJ  1.5     sacred:::ADJ            1.77    25
reverent:::ADJ  1.5     music:::N               4.31    25
reverent:::ADJ  1.5     devout:::ADJ            2.46    25
reverent:::ADJ  1.5     devotion:::N            2.36    25

output_file.txt

reverent:::ADJ  1.5 use:::V         2.78    25
reverent:::ADJ  1.5 sacred:::ADJ    1.77    25
reverent:::ADJ  1.5 music:::N       4.31    25
reverent:::ADJ  1.5 devout:::ADJ    2.46    25
reverent:::ADJ  1.5 devotion:::N    2.36    25
  • Can you give us a small example? – Seth Feb 18 '17 at 16:40
  • You are mixing words and lines together. Consider the line Windows' Eunucchs switched to Unix. The word "Eunuuchs" appears less than 100 times so you delete the line. But you are also deleting "Windows" and "Unix" which might have appeared more than 100 times. Is that what you want to achieve? Like @Seth requested perhaps a small real life example is needed here. – WinEunuuchs2Unix Feb 18 '17 at 16:50
  • you are both right, sorry, I provided an example to make it clearer. I am just considering the strings in column 1 in my specific case, so if the strings in column 1 occur less than 5 times, the whole line should be deleted. – dani_anyman Feb 18 '17 at 17:43
  • Could you give any direction on the file size/number of lines? Is the order of output lines relevant? Are the lines ordered in the sourcefile? – Jacob Vlijm Feb 18 '17 at 17:44
  • @Jacob I edited my post concerning your questions – dani_anyman Feb 18 '17 at 18:05
1

awk with help from process substitution (<()), sort, and uniq:

awk 'NR==FNR{a[$0]; next} {for (i in a) if ($1==i) {print}}' \
      <(awk '{print $1}' file.txt | sort| uniq -c | awk '$1 >= 5 {print $2}') file.txt
  • awk '{print $1}' file.txt | sort| uniq -c | awk '$1 >= 5 {print $2}' gets the first fields where the content occurs greater than or equal to 5 times. The command substitution, <(), replaces the STDOUT with a file descriptor, which is passed as the first argument to the main worker awk process, with the original input file as second

  • NR==FNR{a[$0]; next} creates array a with the elements from first file (NR==FNR) as the keys

  • {for (i in a) if ($1==i) {print}}' prints the lines from file.txt that have the keys from the array a as the first field


The obvious caveat is that this approach is reading the file twice, which might not be the optimum solution for a large (in the usual definition) input file, where high speed and/or low resource usage is expected.


Example:

% cat file.txt                                                                                                     
realism:::N     1.33    depth:::N               3.11    341
realism:::N     1.33    problem:::N             2.68    335
realism:::N     1.33    now:::ADV               1.48    335
realism:::N     1.33    life:::N                2.69    334
renowned:::ADJ  1.41    be:::V                  1.85    15760
renowned:::ADJ  1.41    internationally:::ADV   2.23    9134
renowned:::ADJ  1.41    world:::N               4.36    6736
renowned:::ADJ  1.41    most:::ADV              2.38    5482
reverent:::ADJ  1.5     use:::V                 2.78    25
reverent:::ADJ  1.5     sacred:::ADJ            1.77    25
reverent:::ADJ  1.5     music:::N               4.31    25
reverent:::ADJ  1.5     devout:::ADJ            2.46    25
reverent:::ADJ  1.5     devotion:::N            2.36    25

% awk 'NR==FNR{a[$0]; next} {for (i in a) if ($1==i) {print}}' <(awk '{print $1}' file.txt | sort| uniq -c | awk '$1 >= 5 {print $2}') file.txt
reverent:::ADJ  1.5     use:::V                 2.78    25
reverent:::ADJ  1.5     sacred:::ADJ            1.77    25
reverent:::ADJ  1.5     music:::N               4.31    25
reverent:::ADJ  1.5     devout:::ADJ            2.46    25
reverent:::ADJ  1.5     devotion:::N            2.36    25
  • it nevertheless works perfectly, so thank you so much for your help and also the great description! I really appreciate it! – dani_anyman Feb 18 '17 at 19:08
  • You can avoid the named FIFO in favour of an anonymous pipe: awk '{print $1}' file.txt | sort| uniq -c | awk '$1 >= 5 {print $2}' | awk 'NR==FNR{a[$0]; next} {for (i in a) if ($1==i) {print}}' - file.txt – David Foerster Feb 19 '17 at 13:30
  • @DavidFoerster Not a big fan of hogging the kernel space with numerous anonymous pipes (although they are the nicest form of IPC). – heemayl Feb 20 '17 at 3:25
  • FIFOs are just references to pipes, so the resource use is pretty much the same if you discount the inode and directory entry for the FIFO. – David Foerster Feb 20 '17 at 4:44
1

Assuming the lines are sorted (grouped)

Since you mention the lines are alphabetically sorted by the first column:
The script below will read the lines, keeps them in buffer as long as the line starts with the same string in column one as the previous one.

If not, The buffer is added to an output file (only) if the number of strings meets a certain value, the buffer is subsequently cleared and the process starts again, until the file is done.

This should be pretty fast on large files, although I didn't run a time test (yet) on huge files

The script

#!/usr/bin/env python3
import sys

#-- set the minimum number below
n = 5
# don't change anything below
f = sys.argv[1]; out = sys.argv[2]; mark1 = ""; lines = []

def write_out(lines):
    if len(lines) >= n:
        with open(out, "a+") as wrt:
            for line in lines:
                wrt.write(line)

with open(f) as read:
    for l in read:
        mark2 = l.split()[0]
        if mark2 == mark1:
            lines.append(l)
        else:
            write_out(lines)
            lines = [l]
        mark1 = mark2
# add the last set of lines
write_out(lines)

To use

  1. Copy the script into an empty file, save it as get_lines.py
  2. In the head of the script, set the minimum number of lines, starting with the string.
  3. Run it with the input- and output file as arguments:

    python3 /path/to/get_lines.py <input_file> <output_file>
    

Output (as expected) in output_file, if we set the limit to 5:

reverent:::ADJ  1.5     use:::V                 2.78    25
reverent:::ADJ  1.5     sacred:::ADJ            1.77    25
reverent:::ADJ  1.5     music:::N               4.31    25
reverent:::ADJ  1.5     devout:::ADJ            2.46    25
reverent:::ADJ  1.5     devotion:::N            2.36    25
  • great, thank you also for the detailed description to use the script! It is perfect! Thank you so much! – dani_anyman Feb 18 '17 at 19:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.