consider this file:

#!/usr/bin/env bash
cat > example_file.txt <<EOL
group, value
1, 3.21
1, 3.42
1, 3.5
2, 4.1
2, 4.2

in the following script, I group the rows of this file by the values in the first column (the values in the first column are already sorted) and print each group to an individual txt file:

var=$(echo 'example_file.txt')
var2=$(echo $var|sed "s/.txt//g")
mkdir -p output
cat $var | awk -v varn="$var2" -F, 'FNR == 1 {header = $0;next} !seen[$1]++ { print header > ("output/"varn"_"$1".txt") }{print > ("output/"varn"_"$1".txt");}'


How to print the result to a compressed stream "output/"varn"_"$1".gz" (instead of an uncompressed txt file "output/"varn"_"$1".txt")?

(so the desired output is the same as that the scrip produces now, only I want the outputed files to be compressed and saved to .txt.gz instead of plain text ones as the code does now).

(I tried using gzip > inside the {print} blocks but to no avail :(

(PS I'm a bit of a an awk noob and so the question might be a really dumb one.)

  • 1
    FYI that's a UUOC(Useless Use of Cat) and also of echo and sed: you can assign the variables simply as var='example_file.txt' and varn="{var%.txt}" – steeldriver Aug 18 '18 at 10:42

You can pipe to commands in GNU awk's print. From the GNU awk manual:

print items | command

It is possible to send output to another program through a pipe instead of into a file. This redirection opens a pipe to command, and writes the values of items through this pipe to another process created to execute command.

The redirection argument command is actually an awk expression. Its value is converted to a string whose contents give the shell command to be run. For example, the following produces two files, one unsorted list of peoples’ names, and one list sorted in reverse alphabetical order:

awk '{ print $1 > "names.unsorted"
       command = "sort -r > names.sorted"
       print $1 | command }' mail-list


awk -v varn="$var2" -F, 'FNR == 1 {header = $0;next}
  !seen[$1]++ { print header | "gzip > "output/"varn"_"$1".gz" }
  {print | "gzip > output/"varn"_"$1".gz";}'

For example:

% echo 1 2 | awk '{print $2 | "gzip > "$1".gz"}'
% zcat 1.gz 
  • Thanks! Small problem (I think) running the corrected code (copy pasting the block after the So:) I get awk: 1: unexpected character '.' awk: line 1: runaway string constant ");} ... – user2413 Aug 19 '18 at 17:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.