consider this file:

#!/usr/bin/env bash
cat > example_file.txt <<EOL
group, value
1, 3.21
1, 3.42
1, 3.5
2, 4.1
2, 4.2
EOL

in the following script, I group the rows of this file by the values in the first column (the values in the first column are already sorted) and print each group to an individual txt file:

var=$(echo 'example_file.txt')
var2=$(echo $var|sed "s/.txt//g")
mkdir -p output
cat $var | awk -v varn="$var2" -F, 'FNR == 1 {header = $0;next} !seen[$1]++ { print header > ("output/"varn"_"$1".txt") }{print > ("output/"varn"_"$1".txt");}'

question

How to print the result to a compressed stream "output/"varn"_"$1".gz" (instead of an uncompressed txt file "output/"varn"_"$1".txt")?

(so the desired output is the same as that the scrip produces now, only I want the outputed files to be compressed and saved to .txt.gz instead of plain text ones as the code does now).

(I tried using gzip > inside the {print} blocks but to no avail :(

(PS I'm a bit of a an awk noob and so the question might be a really dumb one.)

  • 1
    FYI that's a UUOC(Useless Use of Cat) and also of echo and sed: you can assign the variables simply as var='example_file.txt' and varn="{var%.txt}" – steeldriver Aug 18 at 10:42
up vote 6 down vote accepted

You can pipe to commands in GNU awk's print. From the GNU awk manual:

print items | command

It is possible to send output to another program through a pipe instead of into a file. This redirection opens a pipe to command, and writes the values of items through this pipe to another process created to execute command.

The redirection argument command is actually an awk expression. Its value is converted to a string whose contents give the shell command to be run. For example, the following produces two files, one unsorted list of peoples’ names, and one list sorted in reverse alphabetical order:

awk '{ print $1 > "names.unsorted"
       command = "sort -r > names.sorted"
       print $1 | command }' mail-list

So:

awk -v varn="$var2" -F, 'FNR == 1 {header = $0;next}
  !seen[$1]++ { print header | "gzip > "output/"varn"_"$1".gz" }
  {print | "gzip > output/"varn"_"$1".gz";}'

For example:

% echo 1 2 | awk '{print $2 | "gzip > "$1".gz"}'
% zcat 1.gz 
2
  • Thanks! Small problem (I think) running the corrected code (copy pasting the block after the So:) I get awk: 1: unexpected character '.' awk: line 1: runaway string constant ");} ... – user2413 Aug 19 at 17:11

Your Answer

 

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.