curl .gz file and pipe it for decompression

Question

I am trying to download some .gz files (N.B. not .tar.gz ones) from given URLs and decompress them to overwrite existing files, if any.

For each individual download, I tried the following:

curl -O $URL | gunzip -f

However, this did not work as it failed with: gzip: stdin: unexpected end of file. I ran a series of this command inside a bash shell script.

If I spilt the command into two explicit steps, i.e. first download the file, and then decompress the .gz file, it works.

Why the piped version does not work?

Answer 1

A pipe (represented by the | symbol) sends the standard output of one process to the standard input of another. In your case, you appear to want to use a named file so a pipe is not appropriate - specifically, there is nothing to pipe (hence the gunzip error) because the remote contents are going to a local file. Instead, you'd need to extract the name of the file - for example, from its URL - something like (using bash's built in string manipulation capabilities)

curl -O "$URL" && gunzip -f "${URL##*/}"

If you want to use a pipe, then the way to do it would be something like

curl "$URL" | gunzip -c

(without the -O option) so that curl streams the remote contents to stdout from where it can be piped into gunzip, but then you would need to redirect the gunzip output to overwrite the target uncompressed file as appropriate.

Answer 2

Follow redirects when downloading. Sometimes a web server has hidden redirects for security and/or random reasons. If you don't follow the redirect, the wrong data gets downloaded and your application reading the piped data gets confused. You can follow redirects with curl using the -L flag.

curl -L https://example.com/mygzip.tar.gz | tar zxv

Answer 3

The below does 3 things

    mkdir -p ${DIR} && curl -L http://apachemirror.wuchna.com/spark/spark-2.4.5/spark-2.4.5-bin-hadoop2.7.tgz | tar zxv -C ${DIR} --strip-components 1

1. Force creates the destination directory
2. UnArchives the archive to the given directory
3. Replaces the root directory with the given directory name by simply stripping down the archive's root directory.