17

I have a file containing approximately 10 million lines.

I want to remove all lines in the file that are less than six characters.

How do I do this?

  • Isn't this question more suited for Stackoverflow? – user1073075 Jan 28 '16 at 19:51
  • 2
    @user1073075 it is perfectly on-topic here. – Seth Feb 3 '16 at 0:46
30

There are many ways to do this.

Using grep:

grep -E '^.{6,}$' file.txt >out.txt

Now out.txt will contain lines having six or more characters.

Reverse way:

grep -vE '^.{,5}$' file.txt >out.txt

Using sed, removing lines of length 5 or less:

sed -r '/^.{,5}$/d' file.txt

Reverse way, printing lines of length six or more:

sed -nr '/^.{6,}$/p' file.txt 

You can save the output in a different file using > operator like grep or edit the file in-place using -i option of sed:

sed -ri.bak '/^.{6,}$/' file.txt 

The original file will be backed up as file.txt.bak and the modified file will be file.txt.

If you do not want to keep a backup:

sed -ri '/^.{6,}$/' file.txt

Using shell, Slower, Don't do this, this is just for the sake of showing another method:

while IFS= read -r line; do [ "${#line}" -ge 6 ] && echo "$line"; done <file.txt

Using python,even slower than grep, sed:

#!/usr/bin/env python2
with open('file.txt') as f:
    for line in f:
        if len(line.rstrip('\n')) >= 6:
            print line.rstrip('\n')

Better use list comprehension to be more Pythonic:

#!/usr/bin/env python2
with open('file.txt') as f:
     strip = str.rstrip
     print '\n'.join([line for line in f if len(strip(line, '\n')) >= 6]).rstrip('\n')
  • Yay! I was hoping for a python answer =) – TellMeWhy Jan 27 '16 at 12:46
  • @DevRobot I see..then check out list comprehension i added, be more Pythonic.. – heemayl Jan 27 '16 at 12:49
  • 1
    Also @DevRobot not so sure python is slower on huge files, when the first option is used. Actually I am pretty sure python is faster on millions of lines, since it reads per line. – Jacob Vlijm Jan 27 '16 at 14:55
  • 1
    The second python example reads the whole file into memory before doing the join. I think the first python example is better in this instance. – Holloway Jan 27 '16 at 23:09
  • Reading by lines is necessarily slower because files are not structured like that. You need to read a block ahead anyway and search for a newline with reduced possibilities of parallelization, then return only the partial string. You need a circular buffer. You need to allocate memory dynamically if you don't know how long the lines can be. – The Vee Jan 28 '16 at 12:28
19

It's very simple:

grep ...... inputfile > resultfile   #There are 6 dots

This is extremely efficient, as grep will not try to parse more than it needs, nor to interpret the chars in any way: it simply send a (whole) line to stdout (which the shell then redirects to resultfile) as soon as it saw 6 chars on that line (. in a regexp context matches any 1 character).

So grep will only output lines having 6 (or more) chars, and the other ones are not outputted by grep so they don't make it to resultfile.

14

Solution #1: using C

Fastest way: compile and run this C program:

#include <stdio.h>
#include <stdlib.h>
#include <string.h>

#define MAX_BUFFER_SIZE 1000000

int main(int argc, char *argv[]) {
    int length;

    if(argc == 3)
        length = atoi(argv[2]);
    else
        return 1;

    FILE *file = fopen(argv[1], "r");

    if(file != NULL) {
        char line[MAX_BUFFER_SIZE];

        while(fgets(line, sizeof line, file) != NULL) {
            char *pos;

            if((pos = strchr(line, '\n')) != NULL)
                *pos = '\0';
            if(strlen(line) >= length)
                printf("%s\n", line);
        }

        fclose(file);
    }
    else {
        perror(argv[1]);
        return 1;
    }

    return 0;
}

Compile with gcc program.c -o program, run with ./program file line_length (where file = path to the file and line_length = minimum line length, in your case 6; the maximum line length is limited to 1000000 characters per line; you can change this by changing the value of MAX_BUFFER_SIZE).

(Trick to substitute \n with \0 found here.)

Comparison with all the other solutions proposed to this question except the shell solution (test run on a ~91MB file with 10M lines with an average lenght of 8 characters):

time ./foo file 6

real    0m1.592s
user    0m0.712s
sys 0m0.160s

time grep ...... file

real    0m1.945s
user    0m0.912s
sys 0m0.176s

time grep -E '^.{6,}$'

real    0m2.178s
user    0m1.124s
sys 0m0.152s

time awk 'length>=6' file

real    0m2.261s
user    0m1.228s
sys 0m0.160s

time perl -lne 'length>=6&&print' file

real    0m4.252s
user    0m3.220s
sys 0m0.164s

sed -r '/^.{,5}$/d' file >out

real    0m7.947s
user    0m7.064s
sys 0m0.120s

./script.py >out
real    0m8.154s
user    0m7.184s
sys 0m0.164s

Solution #2: using AWK:

awk 'length>=6' file
  • length>=6: if length>=6 returns TRUE, prints the current record.

Solution #3: using Perl:

perl -lne 'length>=6&&print' file
  • If lenght>=6 returns TRUE, prints the current record.

% cat file
a
bb
ccc
dddd
eeeee
ffffff
ggggggg
% ./foo file 6
ffffff
ggggggg
% awk 'length>=6' file   
ffffff
ggggggg
% perl -lne 'length>=6&&print' file
ffffff
ggggggg
  • 1
    Believe me..I was waiting for your awk solution.. – heemayl Jan 27 '16 at 12:38
  • 2
    @heemayl And I didn't saw the question immediately, so I knew that if you happened to be online you would have been faster. Had to delete my sed solution (it happens, I know). XD – kos Jan 27 '16 at 12:44
  • What is the point of the pos variable? I get it returns a pointer to the character in line with a newline character, but you never seem to use it. And if you don't find it, you just set it equal to \0. – user1717828 Jan 27 '16 at 17:13
  • @user1717828 If I find it I replace it with \0 (strchr() returns a NULL pointer if the character isn't found). The point is replacing each newline at the end of each line with \0 so that the newline is never counted by strlen(): this is so that the lenght can always be compared to 6 regardless of a potential missing newline on the last line. Treating differently only the last line would be way more efficient, I know. I'll probably update this later. – kos Jan 27 '16 at 17:44
  • 1
    @tripleee The idea was to add a solution useful for something more than a one-time job, or for even bigger files, but: I tested the grep solution on the same file and it's actually faster (probably because strlen() is not the best idea here). I'll try to use a getchar() loop in order to check only the first N character instead, I guess that should improve it visibly. And yes, any line over the buffer's lenght is simply cut to the buffer's length. – kos Jan 28 '16 at 8:35
2

You can use Vim in Ex mode:

ex -sc 'v/\v.{6}/d' -cx file
  1. \v turn on magic

  2. .{6} find lines with 6 or more characters

  3. v invert selection

  4. d delete

  5. x save and close

1

Ruby solution:

$ cat input.txt                                                                                                          
abcdef
abc
abcdefghijk

$ ruby -ne 'puts $_ if $_.chomp.length() >= 6 ' < input.txt                                                              
abcdef
abcdefghijk

Simple idea: redirect file into ruby's stdin, and print line from stdin only if it's length greater or equal to 6

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.