131

How can I download files (that are listed in a text file) using wget or some other automatic way?

Sample file list:

www.example.com/1.pdf
www.example.com/2.pdf
www.example.com/3.pdf
228

wget has a built-in flag for this: wget -i your_list You can find this kind of thing by reading man wget

  • @aureianimus if I want to skip the link 2 then how? – alhelal Nov 22 '17 at 13:11
  • This solution worked for me. I was happy with it downloading sequentially and not in parallel. – asiby Dec 21 '17 at 16:14
85

Get them in parallel with

cat urlfile | parallel --gnu "wget {}"

By default it will run as many processes as you have cores, you can probably ramp this up another 10x if you really want to pull them down quickly by adding "-j 20" after parallel.

  • 1
    I for one just can't get it to work. I don't see any proc spawned, switching echo for wget doesn't output anything – Jakub Bochenski May 9 '14 at 19:10
  • Some earlier ubuntus have a dumb defect in this space: stackoverflow.com/questions/16448887/… – meawoppl May 11 '14 at 2:30
  • 2
    Note with the 'it will run as many processes as you have cores' - network bandwidth is likely going to be more of a limiting factor. – Wilf Jun 21 '14 at 17:10
  • 2
    It really depends. For a large number of small files this can be almost an order of magnitude faster, as most of the transfer time is the handshake/TCP round trip's. Also in the situation where you are downloading from a number of smaller hosts, sometime the per connection bandwidth is limited, so this will bump things up. – meawoppl Jun 23 '14 at 17:22
  • 2
    This is pretty useful if you want to use a list of relative URLs (resource ID without hostnames) with different hostnames, example: cat urlfile | parallel --gnu "wget example1.com{}" and cat urlfile | parallel --gnu "wget example2.com{}" – Mauricio Sánchez May 14 '15 at 2:21
10
xargs -i wget 'http://{}'  < your_list
9

parallel has a built-in flag --arg-file (-a) that will use an input-file as the source, so you can avoid cat |. You can use

parallel --gnu -a urlfile wget

Or simply parallel --gnu wget < urlfile

5
awk '{print "http://" $0;}' list.txt | xargs -l1 wget

where list.txt is your list file

4

I saw Florian Diesch's answer.

I got it to work by including the parameter bqc in the command.

xargs -i wget -bqc 'http://{}' < download.txt

All downloads started in parallel in the background.

  • -b: Background. Go to background immediately after start
  • -q: Quiet. Turn off wget's output
  • -c: Continue. Continue getting a partially-downloaded file
1

Link file links.txt

Command for down load all links file

cat links.txt | wget -i
  • 3
    This does not work. wget -i links.txt is the right command. – Hery Jul 8 '18 at 0:34
  • No this is not right command. Right command is this : " cat links.txt | wget -i" – DreamCoder Jul 19 '18 at 5:07
0

I just tested this:

xargs -a download_file -L1 wget

It works for me. Links inside the txt file must be in separate lines.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.