Take the 2-minute tour ×
Ask Ubuntu is a question and answer site for Ubuntu users and developers. It's 100% free, no registration required.

I have been using this command:

cat urls.txt | xargs -n 1 -P 10 wget -q

to download text files of URLs. This worked fine when my URL file was like:

http://domain1.com/page.html
http://domain2.com/page.html
http://domain3.com/page.html

However I now need to download text files of URLs and post data, like:

--post-data '1=1' http://domain1.com/page.html
--post-data '1=1' http://domain2.com/page.html
--post-data '1=1' http://domain3.com/page.html

When using the above cat command it tries to download the URL and then the post data as a URL. e.g. in the above example, it would download http://domain1.com/page.html and then try and download --post-data 1=1, then http://domain2.com/page.html and so on.

Is there anyway to get cat to send each line of the URLs files only?

Update: I've found that by adding an escape to the space like:

--post-data '1=1'\ http://domain1.com/page.html

is making it be treated as one url, but the -- appear to be stripped from the --post-data argument.

share|improve this question
3  
Try -L 1 instead of -n 1. This will run wget once for each line of input rather than once for each word of input. –  Mark Plotnick Jan 23 at 0:09
    
Thank you Mark Plotnick, that has done the trick. Feel free to post the answer if you would like the rep. from this post. –  Kohjah Breese Jan 23 at 0:12

3 Answers 3

up vote 3 down vote accepted

To make xargs run your command once for each line of input, give it the -L 1 option (and remove the -n 1 option, since they are mutually exclusive). The standards document for xargs says this:

-L number
The utility shall be executed for each non-empty number lines of arguments
from standard input. A line is considered to end with the first <newline>
unless the last character of the line is a <blank>; a trailing <blank>
signals continuation to the next non-empty line, inclusive.
share|improve this answer

Open the terminal and run:

cat urls.txt | sed "s/[\"\<\>' \t\(\);]/\n/g" | grep "http://" | sort -u | xargs -n 1 -P 10 wget -q
share|improve this answer

You don't need xargs,you can do this with a simple bash loop Either keep your URLs as a simple list and add the options directly to wget:

while IFS= read -r url; do wget --post-data '1=1' "$url"; done < urls.txt

or include them in the file and pass them to wget:

while IFS= read -r url; do wget "$url"; done < urls.txt
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.