16

I am trying to download some .gz files (N.B. not .tar.gz ones) from given URLs and decompress them to overwrite existing files, if any.

For each individual download, I tried the following:

curl -O $URL | gunzip -f

However, this did not work as it failed with: gzip: stdin: unexpected end of file. I ran a series of this command inside a bash shell script.

If I spilt the command into two explicit steps, i.e. first download the file, and then decompress the .gz file, it works.

Why the piped version does not work?

3
  • 1
    Are you sure that curl -O actually streams the file to standard output? Perhaps you are thinking of wget -O-? Oct 18, 2014 at 17:00
  • @steeldriver you should post that as an answer curl | tar xz ...
    – Panther
    Oct 18, 2014 at 17:11
  • 1
    There is also --compressed, but this works only for a compressed responses.
    – kenorb
    May 8, 2015 at 23:11

3 Answers 3

19

A pipe (represented by the | symbol) sends the standard output of one process to the standard input of another. In your case, you appear to want to use a named file so a pipe is not appropriate - specifically, there is nothing to pipe (hence the gunzip error) because the remote contents are going to a local file. Instead, you'd need to extract the name of the file - for example, from its URL - something like (using bash's built in string manipulation capabilities)

curl -O "$URL" && gunzip -f "${URL##*/}"

If you want to use a pipe, then the way to do it would be something like

curl "$URL" | gunzip -c

(without the -O option) so that curl streams the remote contents to stdout from where it can be piped into gunzip, but then you would need to redirect the gunzip output to overwrite the target uncompressed file as appropriate.

8

Follow redirects when downloading. Sometimes a web server has hidden redirects for security and/or random reasons. If you don't follow the redirect, the wrong data gets downloaded and your application reading the piped data gets confused. You can follow redirects with curl using the -L flag.

curl -L https://example.com/mygzip.tar.gz | tar zxv
0

The below does 3 things

    mkdir -p ${DIR} && curl -L http://apachemirror.wuchna.com/spark/spark-2.4.5/spark-2.4.5-bin-hadoop2.7.tgz | tar zxv -C ${DIR} --strip-components 1
  1. Force creates the destination directory
  2. UnArchives the archive to the given directory
  3. Replaces the root directory with the given directory name by simply stripping down the archive's root directory.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.