8

I want to print all the lines except the last three lines from the input through awk only. Please note that my file contains n number of lines.

For example,

file.txt contains,

foo
bar
foobar
barfoo
last
line

I want the output to be,

foo
bar
foobar

I know it could be possible through the combination of tac and sed or tac and awk

$ tac file | sed '1,3d' | tac
foo
bar
foobar

$ tac file | awk 'NR==1{next}NR==2{next}NR==3{next}1' | tac
foo
bar
foobar

But i want the output through awk only.

2
  • 3
    one easy way: head -n -3 file.txt
    – sourav c.
    Jun 2, 2014 at 11:59
  • @souravc yep, i know :-) Jun 2, 2014 at 12:00

4 Answers 4

17

It's ever-so clunky but you can add every line to an array and at the end —when you know the length— output everything but the last 3 lines.

... | awk '{l[NR] = $0} END {for (i=1; i<=NR-3; i++) print l[i]}'

Another (more efficient here) approach is manually stacking in three variables:

... | awk '{if (a) print a; a=b; b=c; c=$0}'

a only prints after a line has moved from c to b and then into a so this limits it to three lines. The immediate upsides are it doesn't store all the content in memory and it shouldn't cause buffering issues (fflush() after printing if it does) but the downside here is it's not simple to scale this up. If you want to skip the last 100 lines, you need 100 variables and 100 variable juggles.

If awk had push and pop operators for arrays, it would be easier.

Or we could pre-calculate the number of lines and how far we actually want to go with $(($(wc -l < file) - 3)). This is relatively useless for streamed content but on a file, works pretty well:

awk -v n=$(($(wc -l < file) - 3)) 'NR<n' file

Typically speaking you'd just use head though:

$ seq 6 | head -n-3
1
2
3

Using terdon's benchmark we can actually see how these compare. I thought I'd offer a full comparison though:

  • head: 0.018s (me)
  • awk + wc: 0.169s (me)
  • awk 3 variables: 0.178s (me)
  • awk double-file: 0.322s (terdon)
  • awk circular buffer: 0.355s (Scrutinizer)
  • awk for-loop: 0.693s (me)

The fastest solution is using a C-optimised utility like head or wc handle the heavy lifting things but in pure awk, the manually rotating stack is king for now.

8
  • 4
    +1, head is the best choice. Jun 2, 2014 at 13:04
  • 2
    @glennjackman, but the OP asked for awk only. Besides, head used with -n-3 is a GNU extension and is not specified by POSIX . Jun 2, 2014 at 13:47
  • 4
    Yes, it's a GNU extension. However, this is Ask Ubuntu, so I don't think that's a problem ... Jun 2, 2014 at 13:49
  • 1
    @glennjackman Ow, yes, that's right :) Jun 2, 2014 at 13:52
  • 1
    +1 for matching your username with your behaviour ;) Jun 2, 2014 at 14:01
6

For minimal memory usage, you could use a circular buffer:

awk 'NR>n{print A[NR%n]} {A[NR%n]=$0}' n=3 file

By using the mod operator on line numbers we have at most n array entries.

Taking the example of n=3:

On line 1 NR%n equals 1, line 2 produces 2 and line 3 produces 0 and line 4 evaluates to 1 again..

Line 1 -> A[1]
Line 2 -> A[2]
Line 3 -> A[0]
Line 4 -> A[1]
Line 5 -> A[2]
...

When we get to line 4, A[NR%n] contains the content of line 1. So that gets printed and A[NR%n] gets the content of line 4. The next line (line 5) the original content of line 2 gets printed and so on, until we get to the end. What remains unprinted is the content of the buffer, which contains the last 3 lines...

4
  • works. It would be better if you explain your command little bit. Jun 2, 2014 at 12:30
  • Hi, @AvinashRaj: Added an explanation.. Jun 2, 2014 at 12:41
  • Why'd you delete this? I like it.
    – Oli
    Jun 2, 2014 at 15:08
  • Hi @Oli , I had the impression that the OP was happy with the GNU head solution, which in Ubuntu is available, so I thought my post was not so relevant.. Jun 2, 2014 at 17:00
2

You can also process the file twice to avoid keeping anything in memory:

awk '{if(NR==FNR){c++}else if(FNR<=c-3){print}}' file file

The trick here is the NR==FNR test. NR is the current line number and FNR is the current line number of the current file. If more than one file is passed as input, FNR will be equal to NR only while the first file is being processed. This way, we quickly get the number of lines in the first file and save it as c. Since the "two" files are actually the same one, we now know the number of lines we want so we only print if this is one of them.

While you might think this will be slower than the other approaches, it is actually faster since there is next to no processing going on. Everything is done using internal awk tools (NR and FNR) apart from a single arithmetic comparison. I tested on a 50MB file with one million lines created with this command:

for i in {500000..1000000}; do 
    echo "The quick brown fox jumped over the lazy dog $i" >> file; 
done

As you can see, the times are almost identical but the approach I provided here is marginally faster Oli's first suggestion (but slower than the others):

$ for i in {1..10}; do ( 
    time awk '{if(NR==FNR){c++}else if(FNR<=c-3){print}}' file file > /dev/null ) 2>&1 | 
       grep -oP 'real.*?m\K[\d\.]+'; 
  done | awk '{k+=$1}END{print k/10" seconds"}'; 
0.4757 seconds

$  for i in {1..10}; do ( 
    time awk '{l[NR] = $0} END {for (i=1; i<=NR-3; i++) print l[i]}' file > /dev/null ) 2>&1 | 
        grep -oP 'real.*?m\K[\d\.]+'; 
   done | awk '{k+=$1}END{print k/10" seconds"}'; 
0.5347 seconds
6
  • 3
    I was thinking similarly - you could do a slightly golfed version awk 'NR!=FNR {FNR==1?c=NR:1} FNR<c-3' file file using the NR,FNR builtins instead of a separate count (c here is the file length plus one) and default print action. Jun 2, 2014 at 22:23
  • Don't leave your benchmarking there. The circular buffer is as fast as yours and my second approach (three variables) is actually twice as fast as yours.
    – Oli
    Jun 3, 2014 at 7:41
  • @Oli so it is indeed. Sorry, since I was expecting mine to be slower, I only took the first solution from the accepted answer and was pleasantly surprised.
    – terdon
    Jun 3, 2014 at 11:28
  • The cleverness of FNR and NR usage Aug 12, 2015 at 12:40
  • even simpler version: awk 'FNR<=NR-FNR-3' file file; explanation: NR-FNR is the number of lines in files completely processed so far (0 for the first file, and number of lines in the first file for the second one)
    – mik
    Oct 28, 2022 at 17:25
2

I know the question was specifically about awk, but for brevity one could always use:

head -n -3

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .