Take the 2-minute tour ×
Ask Ubuntu is a question and answer site for Ubuntu users and developers. It's 100% free, no registration required.

How can I download files (that are listed in a text file) using wget or some other automatic way?

Sample file list:
www.example.com/1.pdf
www.example.com/2.pdf

share|improve this question

6 Answers 6

wget has a built-in flag for this: wget -i your_list You can find this kind of thing by reading man wget

share|improve this answer

Get them in parallel with

cat urlfile | parallel --gnu "wget {}"

By default it will run as many processes as you have cores, you can probably ramp this up another 10x if you really want to pull them down quickly by adding "-j 20" after parallel.

share|improve this answer
    
I get as many up votes as down for this. No comments, no love! –  meawoppl Apr 16 at 1:29
    
I for one just can't get it to work. I don't see any proc spawned, switching echo for wget doesn't output anything –  Jakub Bochenski May 9 at 19:10
    
Some earlier ubuntus have a dumb defect in this space: stackoverflow.com/questions/16448887/… –  meawoppl May 11 at 2:30
    
I posted a change that should fix the above. –  meawoppl May 20 at 1:15
    
Note with the 'it will run as many processes as you have cores' - network bandwidth is likely going to be more of a limiting factor. –  Wilf Jun 21 at 17:10
awk '{print "http://" $0;}' list.txt | xargs -l1 wget

where list.txt is your list file

share|improve this answer
xargs -i wget 'http://{}'  < your_list
share|improve this answer

Wget is the best downloader I have ever seen.

You can either use some shell programming with loops to download files whose names follow a pattern like (1,2,3,...) or use wget features.

see this page on how to use wget for downloading files

share|improve this answer

with a for i loop starting with i=1

and

www.example.com/$i.pdf

on your wget code.

You can do it in a single line.

share|improve this answer
1  
Could you give an example of the code instead of just the idea for it? –  Wilf Jun 21 at 17:11

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.