1

Consider this file:

#!/usr/bin/env bash
cat > example_file.txt <<EOL
sdk,jalxa,9325
m,head1,3
ae,wea,2232
ax1,awq,2321
m,head1,32
dgf,12a,8923
m,head1,11
bsda,awq,902
9as,wea,1732
ax1,awq,2321
EOL

I would like to split it on line begining with m,head1,. So the result would be:

file_wea.txt:

m,head1,3
ae,wea,2232
ax1,awq,2321

file_12a.txt:

m,head1,32
dgf,12a,8923

file_awq.txt:

m,head1,11
bsda,awq,902
9as,wea,1732
ax1,awq,2321

e.g. the files resulting from the split should be named after the entry in the second column (comma separated) of the first row after the split pattern. I'm struggling with this part (how to reference the next line in AWK?)

2
+100

You can read in the next line in awk using getline

Quick'n'dirty illustration (you should really check its return value and act accordingly):

awk -F, '
  /^m,head/ {
    tmp = $0; getline; outfile = "file_" $2 ".txt"; print tmp > outfile
  } 
  outfile != "" {
    print > outfile
  }
' example_file.txt

giving

$ head file_*.txt
==> file_12a.txt <==
m,head1,32
dgf,12a,8923

==> file_awq.txt <==
m,head1,11
bsda,awq,902
9as,wea,1732
ax1,awq,2321

==> file_wea.txt <==
m,head1,3
ae,wea,2232
ax1,awq,2321

If the filenames are not unique, you will probably want to append (>>) rather than truncate (>).


If you want to gzip the chunks, then rather than creating plain text files and then executing gzip via a system call, you can print data via standard input to gzip as follows - taking care that you are Closing Input and Output Redirections

awk -F, '
  /^m,head/ {
    close("gzip >" outfile); 
    tmp = $0; getline; outfile = "file_" $2 ".txt.gz"; print tmp | "gzip >" outfile
  } 
  outfile != "" {
    print | "gzip >" outfile
  } 
  END {
    close("gzip >" outfile)
  }
' example_file.txt
  • Thanks a lot. Between this, your answer to my Sunday question and your comment below my now deleted previous awk question, you have solved my problem! I will bounty as soon as I can! PS In my case, the file names will be unique (but thank you for thinking of the next person to read this answer). – user2413 Sep 23 '19 at 12:28
  • One more question, if I wanted to save them as .txt.gz, and also close the connection (each split string is unique but there are millions of them) I would just need to add END { close(outfile); system("gzip " outfile) } after the last curly bracket, right? – user2413 Sep 23 '19 at 12:32
  • @user2413 adding it as an END rule will only gzip the last file - you will also need to add that to the top of the /^m,head/ rule (so that you close and gzip the current outfile each time before setting its new value). You will need to check that outfile is not the empty string before trying to gzip it. – steeldriver Sep 23 '19 at 12:43
  • 1
    @user2413 I have added an alternate suggestion for gzipping on-the-fly – steeldriver Sep 23 '19 at 13:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.