Pipe to compressed file but also print content to terminal

Question

How do I maintain a readable output from my program while also putting that same output into a compressed file, gzip or otherwise, so I can parse it later?

I run a command that generates a lot of samey lines of clear text that should be easy to compress, but I also want to monitor the output of that command. I know I can use tee to print and then put stdin into a file. But the files are massive (250MB+). I also know that I can use gzip to compress stdin into a file, but then I won't see any output on the screen and gzip's --stdout flag just dumps the compressed data to stdout, which is not what I want.

An example would be

python -c 'for i in range(0, 100): print("A" * 120)' | gzip --best > A.log.gz

but I also want to see the As printed to the console.

Answer 1

You can do this by combining tee and process substitution:

python -c 'for i in range(0, 100): print("A" * 120)' | 
  tee >( gzip > compressed.gz)

The >() construct, which is one kind of process substitution, allows you to treat a command as though it were a file. In this case, a file opened for writing (<() would open for reading). The command above will both print to standard output (since that is what tee does by default) and redirect to the special "file" gzip > compressed.gz meaning it will print to it, thereby passing all output through gzip and finally to the compressed file compressed.gz.