11

If I run wget twice, it doesn't recognise that it has already downloaded that file, and creates a new one. Is there any way to prevent it downloading the file again?

$ wget https://cdn.sstatic.net/askubuntu/img/logo.png
...
Saving to: ‘logo.png’
...

$ wget https://cdn.sstatic.net/askubuntu/img/logo.png
...
Saving to: ‘logo.png.1’
...

(Happy to use curl or a similar scriptable alternative if wget can't do this.)

  • 9
    Well it creates a new one because it DOES recognize the file is there! – nico Jan 13 '15 at 13:46
17

I suggest you use the -N option.

-N
--timestamping
    Turn on time-stamping.

It enables time-stamping, which re-downloads the file only if its newer on the server than the downloaded version.

$ wget -N https://cdn.sstatic.net/askubuntu/img/logo.png
...
Saving to: ‘logo.png’
...

$ wget -N https://cdn.sstatic.net/askubuntu/img/logo.png
...
Server file no newer than local file ‘logo.png’ -- not retrieving.

Caveat (from KasiyA's comment)

If the server is not configured properly, it may always report that the file is new and -N will always re-download the file. In this case, -nc is probably a better option.

  • 3
    When a server is not configured properly -N may fail and wget will always redownload. So sometimes -nc is better than -N – αғsнιη Jan 13 '15 at 13:15
  • 1
    @Kasiy thanks for your comment, there seems to be no good option for all cases. – jofel Jan 13 '15 at 13:26
14

Yes it's -c option.

--continue
    Continue getting a partially-downloaded file.  This is useful when you want to
    finish up a download started by a previous instance of Wget, or by another
    program.

If the file is the same, the second download attempt will stop.

$ wget -c https://cdn.sstatic.net/askubuntu/img/logo.png
...
Saving to: ‘logo.png’
...

$ wget -c https://cdn.sstatic.net/askubuntu/img/logo.png
...
The file is already fully retrieved; nothing to do.

Caveats (from jofel's comments)

If the file has changed on the server, the -c option can give incorrect results.

With -c, wget simply asks the server for any data beyond the part of the already downloaded file, nothing else. It does not check if there was any change in the part of the file that is already downloaded. Thus, you could a corrupted file which is a mixture of the old and new file.


Local test

You can test it by running simple local web-server as following(Thanks to @roadmr's answer):

Open a Terminal windows and type:

cd /path/to/parent-download-dir/
python -m SimpleHTTPServer

Now open another Terminal and do:

wget -c http://localhost:8000/filename-to-download

Note that filename-to-download is the file that located in /path/to/parent-download-dir/ which we want to download it.

Now if you run wget command for multiple times you will see:

The file is already fully retrieved; nothing to do.

Ok,now go to /path/to/parent-download-dir/ directory and add something to the source file, for example if it is a text file, add a simple extra line in it and save the file. Now try with wget -c ... . Great, now you will see the file re-downloads again but you already have downloaded it before.

Reason: why re-downloading?

because its size changed to larger size than old downloaded file and nothing else.

  • 1
    This does not work correctly if the file has changed on the server between the downloads. In the worst case (file size has increased), you get a corrupted file. – jofel Jan 13 '15 at 12:55
  • 1
    @jofel yes -nc doesn't as you said but -c option will work and that's why I mentioned -c option first. – αғsнιη Jan 13 '15 at 12:58
  • With -c, wget asks the server for any data beyond the part of the already downloaded file, nothing else. It does not check if there is meanwhile any change in the already downloaded part of the file on the server. At the worst you get a corrupted file which is a mixture between the old and new file. – jofel Jan 13 '15 at 13:06
  • after reviewing the options I think this is only useful for something like a log file (guaranteed incremental updates), in all other cases I think -N or -nc are more appropriate as they deal with whole files – david.libremone Jan 20 '15 at 7:59
3

Also there is another option called -nc for wgetting:

--no-clobber
   If a file is downloaded more than once in the same directory, Wget's behavior
   depends on a few options, including -nc.  In certain cases, the local file will
   be clobbered, or overwritten, upon repeated download.  In other cases it will be
   preserved.

When -nc option is specified, Wget will refuse to download copies of the same file. If you had the same file that wget tries to download, it will refuse to download it unless you rename or remove the local file.

$ wget -nc https://cdn.sstatic.net/askubuntu/img/logo.png
...
Saving to: ‘logo.png’
...

$ wget -nc https://cdn.sstatic.net/askubuntu/img/logo.png
File ‘logo.png’ already there; not retrieving.

Sometimes this option is strongly good and I recommended to use -nc option instead of both -c or -N option because these options will overwrite the download-file with your local file if they had same names.

Caveat (from jofel's comment)

The -nc option does not update the file if it has changed on the server. If you know the file will change, the -N option is preferable. If you know the file will not change (or you don't care) then -nc is ok.

1

I know this was a specific question concerning wget but the OP did mention "Happy to use curl or a similar scriptable alternative if wget can't do this." I am not sure what the requirement here is (multiple files, keep old version if different from original, replace with newly downloaded version). Depending on what you want and how you want to handle duplicates you may need more than this.. A very simple way to do what you seem to want is simply to use curl instead.

curl http://cdn.sstatic.net/askubuntu/img/logo.png?v=ca4d192163aa > logo.png

This command will replace the old file with the newly downloaded one every time.

Do not output this to the terminal (without the "> [filename]") if you are downloading a binary file as opposed to text. Doing so will could potentially mess with your terminal session. In the case you do do this on accident you may need to open another shell/terminal session.

  • If your terminal is in a bad shape because you displayed binary files, it is maybe easier to call the program "reset" than opening a new terminal. – jofel Jan 14 '15 at 12:49
  • you're right that I wasn't clear on my requirements, the happy result is that I learnt about a few more options :) thanks – david.libremone Jan 20 '15 at 7:54
  • Thanks @jofel ^^ didn't know about "reset" I always just closed opened a new tab and closed the garbled one when this happened... though I haven't really had it happen often. – Goblinlord Jan 21 '15 at 0:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.