How to download big sites with several interrupted wget:s?

Question

I'm trying to download a site with wget, but the site is so big that I can't do it all at once. The problem is that the server is always reporting a new file and when I resume the download wget creates filename.1, filename.2 ... etc., but I know that the files have not changed or if they have I don't care. Is there a way to turn of timestamp checking and make wget assume that a file with the same name is current?

Answer 1

I think you are actually asking how to resume interrupted downloads of big files. In order for that to work, the server must specify in the header how large the file is. Otherwise most tools, it not all, will never resume a download and always start from the beginning, until they no longer receive any bits from the server. You can check manually if the content-length attribute is specified in a header by running curl -I <url.

For example, we can check the header we get for the askubuntu page logo:

$ curl -I https://cdn.sstatic.net/Sites/askubuntu/img/logo.svg

HTTP/2 200 
cache-control: max-age=604800
content-type: image/svg+xml
last-modified: Wed, 22 Jan 2020 23:01:03 GMT
accept-ranges: bytes
date: Thu, 23 Jan 2020 14:30:06 GMT
via: 1.1 varnish
age: 0
x-served-by: cache-mxp19838-MXP
x-cache: MISS
x-cache-hits: 0
x-timer: S1579789807.662934,VS0,VE117
vary: Accept-Encoding
content-length: 11748

At the very bottom you can see content-length: 11748, which would enable wget to resume the download if interrupted.

Also, you can use ariac instead of wget, which in my opinion handles interrupted downloads better.