1

I'm using wget to download some pages and I don't want it to download the same page if it has already been downloaded. I use the following command:

wget --no-clobber --input text04.txt --directory-prefix =/path/file

** the text04.txt file has new links every day that may be the same as in previous days

When downloading the pages they come without the extension (in the case .html). This does not let the file open normally, but if I add the extension after wget downloads when it is executed again, the "--no-clobber" command is useless, since adding the .html is considered by wget with a new file.

Would you like to add the extension (.html) and when the wget is re-executed do not download the same file more than once?

1

You need to add one option --html-extension so the command will be like so:

wget --no-clobber --html-extension --input text04.txt --directory-prefix =/path/file

This way the .html extension will be added automatically.

Please refer to this answer for more information.

Notice:

Unfortunately wget is not capable of adding the .html extension at the checking stage. So it will always re-download the page when --no-clobber and --html-extension options are specified if the remote page has no .html extension but it will at least update and overwrite the existing local copy of the page instead of writing another copy.

Best of luck

3
  • Hi @raffa , thanks for the answer, but it didn't work out. Although with this command (--html-extension) the extension is added to the end of the file when I run wget the files already downloaded are downloaded again
    – Rafael
    Apr 1 '20 at 19:55
  • @Rafael Unfortunately wget is not capable of adding the .html at the checking stage. So it will always re-download the file when --no-clobber and --html-extension options are specified if the remote page has no .html extension but it will at least update and overwrite the existing local copy of the page instead of downloading another copy.
    – Raffa
    Apr 1 '20 at 20:05
  • tank you @raffa
    – Rafael
    Apr 1 '20 at 20:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.