0

I'm trying to download a site with wget, but the site is so big that I can't do it all at once. The problem is that the server is always reporting a new file and when I resume the download wget creates filename.1, filename.2 ... etc., but I know that the files have not changed or if they have I don't care. Is there a way to turn of timestamp checking and make wget assume that a file with the same name is current?

0

I think you are actually asking how to resume interrupted downloads of big files. In order for that to work, the server must specify in the header how large the file is. Otherwise most tools, it not all, will never resume a download and always start from the beginning, until they no longer receive any bits from the server. You can check manually if the content-length attribute is specified in a header by running curl -I <url.

For example, we can check the header we get for the askubuntu page logo:

$ curl -I https://cdn.sstatic.net/Sites/askubuntu/img/logo.svg

HTTP/2 200 
cache-control: max-age=604800
content-type: image/svg+xml
last-modified: Wed, 22 Jan 2020 23:01:03 GMT
accept-ranges: bytes
date: Thu, 23 Jan 2020 14:30:06 GMT
via: 1.1 varnish
age: 0
x-served-by: cache-mxp19838-MXP
x-cache: MISS
x-cache-hits: 0
x-timer: S1579789807.662934,VS0,VE117
vary: Accept-Encoding
content-length: 11748

At the very bottom you can see content-length: 11748, which would enable wget to resume the download if interrupted.

Also, you can use ariac instead of wget, which in my opinion handles interrupted downloads better.

|improve this answer|||||
  • No, not one big file but a website with thousands of small files. I'll look at ariac if I'm getting nowhere with wget, thanks for the pointer. – GenericUser Jan 23 at 14:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.