how to make wget not to download the same file more than once (with and without extension)

Question

I'm using wget to download some pages and I don't want it to download the same page if it has already been downloaded. I use the following command:

wget --no-clobber --input text04.txt --directory-prefix =/path/file

** the text04.txt file has new links every day that may be the same as in previous days

When downloading the pages they come without the extension (in the case .html). This does not let the file open normally, but if I add the extension after wget downloads when it is executed again, the "--no-clobber" command is useless, since adding the .html is considered by wget with a new file.

Would you like to add the extension (.html) and when the wget is re-executed do not download the same file more than once?

Answer 1

You need to add one option --html-extension so the command will be like so:

wget --no-clobber --html-extension --input text04.txt --directory-prefix =/path/file

This way the .html extension will be added automatically.

Please refer to this answer for more information.

Notice:

Unfortunately wget is not capable of adding the .html extension at the checking stage. So it will always re-download the page when --no-clobber and --html-extension options are specified if the remote page has no .html extension but it will at least update and overwrite the existing local copy of the page instead of writing another copy.

Best of luck