Ask Ubuntu is a question and answer site for Ubuntu users and developers. It's 100% free.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

How can I download files (that are listed in a text file) using wget or some other automatic way?

Sample file list:

www.example.com/1.pdf
www.example.com/2.pdf
www.example.com/3.pdf
share|improve this question

Link file links.txt

Command for down load all links file

    cat links.txt | wget -i
share|improve this answer

Sadly, I don't have enough reputation to comment on Florian Diesch's answer, hence this.

I got it to work by including the parameter "bqc" in the command.

xargs -i wget -bqc 'http://{}' < download.txt

All downloads started in parallel in the background.

-b: Background. Go to background immediately after start

-q: Quiet. Turn off wget's output

-c: Continue. Continue getting a partially-downloaded file

share|improve this answer
    
And what do -b, -q and -c do? – muru May 4 '15 at 17:02

Get them in parallel with

cat urlfile | parallel --gnu "wget {}"

By default it will run as many processes as you have cores, you can probably ramp this up another 10x if you really want to pull them down quickly by adding "-j 20" after parallel.

share|improve this answer
    
I get as many up votes as down for this. No comments, no love! – meawoppl Apr 16 '14 at 1:29
1  
I for one just can't get it to work. I don't see any proc spawned, switching echo for wget doesn't output anything – Jakub Bochenski May 9 '14 at 19:10
    
Some earlier ubuntus have a dumb defect in this space: stackoverflow.com/questions/16448887/… – meawoppl May 11 '14 at 2:30
1  
Note with the 'it will run as many processes as you have cores' - network bandwidth is likely going to be more of a limiting factor. – Wilf Jun 21 '14 at 17:10
1  
It really depends. For a large number of small files this can be almost an order of magnitude faster, as most of the transfer time is the handshake/TCP round trip's. Also in the situation where you are downloading from a number of smaller hosts, sometime the per connection bandwidth is limited, so this will bump things up. – meawoppl Jun 23 '14 at 17:22

wget has a built-in flag for this: wget -i your_list You can find this kind of thing by reading man wget

share|improve this answer
xargs -i wget 'http://{}'  < your_list
share|improve this answer
awk '{print "http://" $0;}' list.txt | xargs -l1 wget

where list.txt is your list file

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.