Ask Ubuntu is a question and answer site for Ubuntu users and developers. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have a file containing approximately 10 million lines.

I want to remove all lines in the file that are less than six characters.

How do I do this?

share|improve this question
    
Isn't this question more suited for Stackoverflow? – user1073075 Jan 28 at 19:51
2  
@user1073075 it is perfectly on-topic here. – Seth Feb 3 at 0:46
up vote 29 down vote accepted

There are many ways to do this.

Using grep:

grep -E '^.{6,}$' file.txt >out.txt

Now out.txt will contain lines having six or more characters.

Reverse way:

grep -vE '^.{,5}$' file.txt >out.txt

Using sed, removing lines of length 5 or less:

sed -r '/^.{,5}$/d' file.txt

Reverse way, printing lines of length six or more:

sed -nr '/^.{6,}$/p' file.txt 

You can save the output in a different file using > operator like grep or edit the file in-place using -i option of sed:

sed -ri.bak '/^.{6,}$/' file.txt 

The original file will be backed up as file.txt.bak and the modified file will be file.txt.

If you do not want to keep a backup:

sed -ri '/^.{6,}$/' file.txt

Using shell, Slower, Don't do this, this is just for the sake of showing another method:

while IFS= read -r line; do [ "${#line}" -ge 6 ] && echo "$line"; done <file.txt

Using python,even slower than grep, sed:

#!/usr/bin/env python2
with open('file.txt') as f:
    for line in f:
        if len(line.rstrip('\n')) >= 6:
            print line.rstrip('\n')

Better use list comprehension to be more Pythonic:

#!/usr/bin/env python2
with open('file.txt') as f:
     strip = str.rstrip
     print '\n'.join([line for line in f if len(strip(line, '\n')) >= 6]).rstrip('\n')
share|improve this answer
    
Yay! I was hoping for a python answer =) – DevRobot Jan 27 at 12:46
    
@DevRobot I see..then check out list comprehension i added, be more Pythonic.. – heemayl Jan 27 at 12:49
1  
Also @DevRobot not so sure python is slower on huge files, when the first option is used. Actually I am pretty sure python is faster on millions of lines, since it reads per line. – Jacob Vlijm Jan 27 at 14:55
1  
The second python example reads the whole file into memory before doing the join. I think the first python example is better in this instance. – Holloway Jan 27 at 23:09
    
Reading by lines is necessarily slower because files are not structured like that. You need to read a block ahead anyway and search for a newline with reduced possibilities of parallelization, then return only the partial string. You need a circular buffer. You need to allocate memory dynamically if you don't know how long the lines can be. – The Vee Jan 28 at 12:28

It's very simple:

grep ...... inputfile > resultfile   #There are 6 dots

This is extremely efficient, as grep will not try to parse more than it needs, nor to interpret the chars in any way: it simply send a (whole) line to stdout (which the shell then redirects to resultfile) as soon as it saw 6 chars on that line (. in a regexp context matches any 1 character).

So grep will only output lines having 6 (or more) chars, and the other ones are not outputted by grep so they don't make it to resultfile.

share|improve this answer

Solution #1: using C

Fastest way: compile and run this C program:

#include <stdio.h>
#include <stdlib.h>
#include <string.h>

#define MAX_BUFFER_SIZE 1000000

int main(int argc, char *argv[]) {
    int length;

    if(argc == 3)
        length = atoi(argv[2]);
    else
        return 1;

    FILE *file = fopen(argv[1], "r");

    if(file != NULL) {
        char line[MAX_BUFFER_SIZE];

        while(fgets(line, sizeof line, file) != NULL) {
            char *pos;

            if((pos = strchr(line, '\n')) != NULL)
                *pos = '\0';
            if(strlen(line) >= length)
                printf("%s\n", line);
        }

        fclose(file);
    }
    else {
        perror(argv[1]);
        return 1;
    }

    return 0;
}

Compile with gcc program.c -o program, run with ./program file line_length (where file = path to the file and line_length = minimum line length, in your case 6; the maximum line length is limited to 1000000 characters per line; you can change this by changing the value of MAX_BUFFER_SIZE).

(Trick to substitute \n with \0 found here.)

Comparison with all the other solutions proposed to this question except the shell solution (test run on a ~91MB file with 10M lines with an average lenght of 8 characters):

time ./foo file 6

real    0m1.592s
user    0m0.712s
sys 0m0.160s

time grep ...... file

real    0m1.945s
user    0m0.912s
sys 0m0.176s

time grep -E '^.{6,}$'

real    0m2.178s
user    0m1.124s
sys 0m0.152s

time awk 'length>=6' file

real    0m2.261s
user    0m1.228s
sys 0m0.160s

time perl -lne 'length>=6&&print' file

real    0m4.252s
user    0m3.220s
sys 0m0.164s

sed -r '/^.{,5}$/d' file >out

real    0m7.947s
user    0m7.064s
sys 0m0.120s

./script.py >out
real    0m8.154s
user    0m7.184s
sys 0m0.164s

Solution #2: using AWK:

awk 'length>=6' file
  • length>=6: if length>=6 returns TRUE, prints the current record.

Solution #3: using Perl:

perl -lne 'length>=6&&print' file
  • If lenght>=6 returns TRUE, prints the current record.

% cat file
a
bb
ccc
dddd
eeeee
ffffff
ggggggg
% ./foo file 6
ffffff
ggggggg
% awk 'length>=6' file   
ffffff
ggggggg
% perl -lne 'length>=6&&print' file
ffffff
ggggggg
share|improve this answer
1  
Believe me..I was waiting for your awk solution.. – heemayl Jan 27 at 12:38
2  
@heemayl And I didn't saw the question immediately, so I knew that if you happened to be online you would have been faster. Had to delete my sed solution (it happens, I know). XD – kos Jan 27 at 12:44
    
What is the point of the pos variable? I get it returns a pointer to the character in line with a newline character, but you never seem to use it. And if you don't find it, you just set it equal to \0. – user1717828 Jan 27 at 17:13
    
@user1717828 If I find it I replace it with \0 (strchr() returns a NULL pointer if the character isn't found). The point is replacing each newline at the end of each line with \0 so that the newline is never counted by strlen(): this is so that the lenght can always be compared to 6 regardless of a potential missing newline on the last line. Treating differently only the last line would be way more efficient, I know. I'll probably update this later. – kos Jan 27 at 17:44
1  
@tripleee The idea was to add a solution useful for something more than a one-time job, or for even bigger files, but: I tested the grep solution on the same file and it's actually faster (probably because strlen() is not the best idea here). I'll try to use a getchar() loop in order to check only the first N character instead, I guess that should improve it visibly. And yes, any line over the buffer's lenght is simply cut to the buffer's length. – kos Jan 28 at 8:35

You can use Vim in Ex mode:

ex -sc 'v/\v.{6}/d' -cx file
  1. \v turn on magic

  2. .{6} find lines with 6 or more characters

  3. v invert selection

  4. d delete

  5. x save and close

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.