0

I try to download all the URL from a specific file : file.csv in order to do that I use this command line : wget -c -i file.csv But I would like to get a list of all the URL that wget was not able to download.

When I use this kind of command line : wget -c -i file.csv 1>output_file.txt 2>error_file.txt I got a error_file.txt with to many information and a lot of those information are useless for instance, the URL already downloaded are mentionned in this file (it is not a failure case).

Best,

  • 1
    Use it like this: wget -c -i file.csv --rejected-log=logfile – George Udosen Dec 24 '17 at 11:01
0

You can try that command this way:

wget -c --rejected-log=/path/to/logfile -i file.csv 
  • --rejected-log=logfile:

    Logs all URL rejections to logfile as comma separated values. The values include the reason of rejection, the URL and the parent URL it was
    found in.
    

Note: You need to create a log file for use and give the path in that command

See: man wget

|improve this answer|||||
  • When I try to use : 'wget -c --rejected-log=logfile -i test.csv' with the test.csv that contain only one url line : commons.wikimedia.org/wiki/Special:FilePath/… The jpeg in this url is not found as expected but no logfile is created at all. Do you know why ? – Nicao Dec 24 '17 at 17:42
  • The logfile has to exist! Create one say mylogs.txt, then add it to the line: wget -c --rejected-log=/path/to/mylogs.txt -i file.csv – George Udosen Dec 24 '17 at 18:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.