I'm trying to download a site with wget, but the site is so big that I can't do it all at once. The problem is that the server is always reporting a new file and when I resume the download wget creates filename.1, filename.2 ... etc., but I know that the files have not changed or if they have I don't care. Is there a way to turn of timestamp checking and make wget assume that a file with the same name is current?
I think you are actually asking how to resume interrupted downloads of big files. In order for that to work, the server must specify in the header how large the file is. Otherwise most tools, it not all, will never resume a download and always start from the beginning, until they no longer receive any bits from the server. You can check manually if the
content-length attribute is specified in a header by running
curl -I <url.
For example, we can check the header we get for the askubuntu page logo:
$ curl -I https://cdn.sstatic.net/Sites/askubuntu/img/logo.svg HTTP/2 200 cache-control: max-age=604800 content-type: image/svg+xml last-modified: Wed, 22 Jan 2020 23:01:03 GMT accept-ranges: bytes date: Thu, 23 Jan 2020 14:30:06 GMT via: 1.1 varnish age: 0 x-served-by: cache-mxp19838-MXP x-cache: MISS x-cache-hits: 0 x-timer: S1579789807.662934,VS0,VE117 vary: Accept-Encoding content-length: 11748
At the very bottom you can see
content-length: 11748, which would enable
wget to resume the download if interrupted.
Also, you can use
ariac instead of
wget, which in my opinion handles interrupted downloads better.