0

Consider this file:

#!/usr/bin/env bash
cat > example_file.txt <<EOL
TITLE something
some data
some data
some data
TITLE something else
some other data
TITLE some more
some other data
some other data
some other data
TITLE extra info
some more data
some more data
EOL

I need to add a new column that:

  • counts the number of lines,
  • goes back to 1 after an occurrence of /^TITLE/,
  • starts from the bottom of the file and works its way upward,

Basically, the result should look like:

TITLE something,4
some data,3
some data,2
some data,1
TITLE something else,2
some other data,1
TITLE some more,4
some other data,3
some other data,2
some other data,1
TITLE extra info,3
some more data,2
some more data,1

P.S. you can assume that:

  • the file always starts with a line matching /^TITLE/
  • the file always ends with a line not matching /^TITLE/
  • there are no two consecutive lines matching /^TITLE/

Edit:

Results so far

on a 100MB file:

@Yarom

time tac trial.txt | awk 'BEGIN{x=0} {x++;{if ($1 !~/^pattern/) printf "%s,%s\n",$0,x;else if ($1 ~/^pattern/) {printf "%s,%s\n",$0,x;x=0}}}' | tac > trial2.txt
real    0m0,896s

@bac0n

 time awk '{ a[i++]=$0 } END { while (i--) { a[i]=a[i] "," ++j; if (a[i] ~ /^pattern/) { j=0 } }; for (i=0; i<NR; i++) { print a[i] } }' trial.txt > trial2.txt
real    0m0,830s

@oliv:

time awk -v RS='^pattern' -v FS='\n' '
{
  for(i=NF-1;i>0;i--) 
    printf "%s,%d\n",$i,i; 
    printf RT
}' trial.txt > trial2.txt

real    0m2,343s

@steeldriver

 time awk -vRS='\n(^pattern|$)' -F'\n' -vOFS=, '
   NR>1 {$1 = "^pattern" $1} 
   {for(i=1;i<=NF;i++) print $i, NF-i+1}
 ' trial.txt > trial2.txt
real    0m1,889s

using mawk instead of awk, I get:

mawk: program limit exceeded: maximum number of fields size=32767
  • Sorry for bothering but is there a chance you can also mark it as resolved? Thank you :) – Yaron Sep 24 '19 at 7:42
  • It's not resolved. Reading your answer made me realize I need to edit the question a bit: you cannot assume that the non /^TITLE/ lines start with DATA. – user2413 Sep 24 '19 at 7:47
  • Oops, my bad, I changed it to TITLE and not TITLE, hoping it's better now :) – Yaron Sep 24 '19 at 7:51
  • 1
    thanks @Yaron, works perfect now;) – user2413 Sep 24 '19 at 7:55
  • 1
    awk '{ a[i++]=$0 } END { while (i--) { a[i]=a[i] "," ++j; if (a[i] ~ /^TITLE/) { j=0 } }; for (i=0; i<NR; i++) { print a[i] } }' example_file.txt – bac0n Sep 24 '19 at 8:34
1

example.awk

#!/bin/awk -f

{ a[i++]=$0 } END {
    while (i--) {
        a[i]=a[i] "," ++j
        if (a[i] ~ /^TITLE/) { j=0 }
    }
    for (i in a) { print a[i] }
}

Example

awk -f example.awk example.txt

Output

TITLE something,4
DATA some data,3
DATA some data,2
DATA some data,1
TITLE something else,2
DATA some other data,1
TITLE some more,4
DATA some other data,3
DATA some other data,2
DATA some other data,1
TITLE extra info,3
DATA some more data,2
DATA some more data,1
  • Also does the job perfectly. I will leave the question open for a few hours to see if a faster answer pops up;) – user2413 Sep 24 '19 at 9:19
2

I managed to compile the following one-liner:

tac so_count.txt | awk 'BEGIN{x=0} {x++;{if ($1 != "TITLE") printf "%s,%s\n",$0,x;else if ($1 == "TITLE") {printf "%s,%s\n",$0,x;x=0}}}' | tac

I'll explain a bit further:

  • tac - reverse the order of lines (reverse cat).
  • awk - if the first column is not TITLE advance the counter, if TITLE print the current count and reset back to 0.
  • tac - reverse it back.

Results:

TITLE something,4
DATA some data,3
DATA some data,2
DATA some data,1
TITLE something else,2
DATA some other data,1
TITLE some more,4
DATA some other data,3
DATA some other data,2
DATA some other data,1
TITLE extra info,3
DATA some more data,2
DATA some more data,1

Good luck!

  • your idea gives the desired result but it is very slow. The file is huge, > 10GB. I will leave the question open in the hope someone proposes a faster command. – user2413 Sep 24 '19 at 8:12
  • 1
    OK, that will require actually counting and adding section by section, I can't think of a tool that can do it fast enough, what about splitting the file and running it in parallel on multiple smaller files? (Will probably require lots of RAM). – Yaron Sep 24 '19 at 8:22
  • 1
    How about combining one of these solutions? unix.stackexchange.com/questions/9356/… – Yaron Sep 24 '19 at 8:24
  • 1
    yes, I'm looking at this too. Basically, I'm trying all these solutions. It's already good to know that I have yours in the bank in case these fail. – user2413 Sep 24 '19 at 8:27
2

You could treat each block as a record, and each line as a field - that way, you can get the per-block count down without either reversing the file or loading more than one block into memory.

Because your blocks are delineated by a header rather than a footer, a bit of hackery is required to handle the first an last records. The best I can come up with is:

awk -vRS='\n(TITLE|$)' -F'\n' -vOFS=, '
  NR>1 {$1 = "TITLE" $1} # replace the RS that got stripped off
  {for(i=1;i<=NF;i++) print $i, NF-i+1}
' example_file.txt

This should be valid in both gawk and mawk. I suspect it will run significantly faster in the lower-overhead mawk; gawk speed may be comparable if you set the locale to C/POSIX i.e. LC_ALL=C awk '...'

  • the real case counterpart to TITLE is |H|ENUM_SRC_ID|, should I use \|H\|ENUM_SRC_ID\|? – user2413 Sep 24 '19 at 13:08
  • @user2413 you will need an extra backslash in the external RS regex assignment i.e. -vRS='\n(\\|H\\|ENUM_SRC_ID\\||$)' - the $1 assignment requires no escaping (since it's just a string not a regex) – steeldriver Sep 24 '19 at 13:25
2

Using awk:

awk -v RS='TITLE ' -v FS='\n' '
{
  for(i=NF-1;i>0;i--) 
    printf "%s,%d\n",$i,i; 
    printf RT
}' file

This relies on the record separator RS and the field separator FS that are set to set the right starting value to the counter i.

The only statements print each field with the counter and the record terminator RT associated to RS.

This solution has the advantage to only parse the file once and doesn't require to put the whole file in memory.

  • the real case counterpart to TITLE is |H|ENUM_SRC_ID|, should I use \|H\|ENUM_SRC_ID\|? – user2413 Sep 24 '19 at 13:08
  • @user2413 as | is a keyword regex and awk treats backslash differently, you have to use 2 backslashes : -v RS='\\|H\\|ENUM_SRC_ID\\| ' – oliv Sep 24 '19 at 14:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.