If I run the command s3cmd get s3://bucket/file.gz - I get binary output on the screen. If I try to pipe this to gzip with s3cmd get s3://bucket/file.gz - | zcat I get gzip: stdin: not in gzip format.

How can I get zcat to pick it's input from stdin?


gzip and its auxilliary commands all read from STDIN by default. We can test this with a really simple test:

$ echo testing | gzip | zcat

Or something more exotic to prove that wasn't a fluke:

$ dd if=/dev/urandom of=bigfile bs=1024 count=102400
102400+0 records in
102400+0 records out
104857600 bytes (105 MB, 100 MiB) copied, 6.42114 s, 16.3 MB/s

$ sha1sum bigfile 
25b4832d3e738e70721d86695ea7a767a3afb229  bigfile

$ cat bigfile | gzip | zcat | sha1sum 
25b4832d3e738e70721d86695ea7a767a3afb229  -

That suggests to me that your s3cmd output is dirty or malformed in some way. Try redirecting to file (rather tha providing a real filename) and then looking at the output in something like head. Or download it properly and compare.

  • I think you're right about the s3cmd output. I changed my approach and used curl instead. – KalenGi Mar 11 '16 at 16:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.