163

How can I download files (that are listed in a text file) using wget or some other automatic way?

Sample file list: 

www.example.com/1.pdf
www.example.com/2.pdf
www.example.com/3.pdf
Improve this question

8 Answers 8

Reset to default
273

wget has a built-in flag for this: wget -i your_list, where your_list is a file containing URL's delimited by linebreaks. You can find this kind of thing by reading man wget

Improve this answer
2
  • @aureianimus if I want to skip the link 2 then how?
    – alhelal
    Nov 22, 2017 at 13:11
  • This solution worked for me. I was happy with it downloading sequentially and not in parallel.
    – asiby
    Dec 21, 2017 at 16:14
91

Get them in parallel with

cat urlfile | parallel --gnu "wget {}"

By default it will run as many processes as you have cores, you can probably ramp this up another 10x if you really want to pull them down quickly by adding "-j 20" after parallel.

Improve this answer
14
  • 1
    I for one just can't get it to work. I don't see any proc spawned, switching echo for wget doesn't output anything
    – Jakub Bochenski
    May 9, 2014 at 19:10
  • 2
    Note with the 'it will run as many processes as you have cores' - network bandwidth is likely going to be more of a limiting factor.
    – Wilf
    Jun 21, 2014 at 17:10
  • 2
    It really depends. For a large number of small files this can be almost an order of magnitude faster, as most of the transfer time is the handshake/TCP round trip's. Also in the situation where you are downloading from a number of smaller hosts, sometime the per connection bandwidth is limited, so this will bump things up.
    – meawoppl
    Jun 23, 2014 at 17:22
  • 2
    This is pretty useful if you want to use a list of relative URLs (resource ID without hostnames) with different hostnames, example: cat urlfile | parallel --gnu "wget example1.com{}" and cat urlfile | parallel --gnu "wget example2.com{}"
    – Mauricio Sánchez
    May 14, 2015 at 2:21
  • 1
    One might add that flooding a website for a massive amount of parallel requests for large files is not particularly nice. Doesn't matter for big sites, but if it's just a smaller one you should take care.
    – Magnus
    Sep 19, 2019 at 10:02
14

parallel has a built-in flag --arg-file (-a) that will use an input-file as the source, so you can avoid cat |. You can use

parallel --gnu -a urlfile wget

Or simply parallel --gnu wget < urlfile

Improve this answer
10 
xargs -i wget 'http://{}'  < your_list
Improve this answer
5 
awk '{print "http://" $0;}' list.txt | xargs -l1 wget

where list.txt is your list file

Improve this answer
5

I saw Florian Diesch's answer.

I got it to work by including the parameter bqc in the command.

xargs -i wget -bqc 'http://{}' < download.txt

All downloads started in parallel in the background.

  • -b: Background. Go to background immediately after start
  • -q: Quiet. Turn off wget's output
  • -c: Continue. Continue getting a partially-downloaded file
Improve this answer
0
1

Link file links.txt

Command for down load all links file

cat links.txt | wget -i
Improve this answer
3
  • 5
    This does not work. wget -i links.txt is the right command.
    – Hery
    Jul 8, 2018 at 0:34
  • No this is not right command. Right command is this : " cat links.txt | wget -i"
    – Bhagirathsinh Gohil
    Jul 19, 2018 at 5:07
  • there is a dash missing
    – Summer-Sky
    Feb 12, 2020 at 17:14
0

I just tested this:

xargs -a download_file -L1 wget

It works for me. Links inside the txt file must be in separate lines.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .