3

I have been using this command:

cat urls.txt | xargs -n 1 -P 10 wget -q

to download text files of URLs. This worked fine when my URL file was like:

http://domain1.com/page.html
http://domain2.com/page.html
http://domain3.com/page.html

However I now need to download text files of URLs and post data, like:

--post-data '1=1' http://domain1.com/page.html
--post-data '1=1' http://domain2.com/page.html
--post-data '1=1' http://domain3.com/page.html

When using the above cat command it tries to download the URL and then the post data as a URL. e.g. in the above example, it would download http://domain1.com/page.html and then try and download --post-data 1=1, then http://domain2.com/page.html and so on.

Is there anyway to get cat to send each line of the URLs files only?

Update: I've found that by adding an escape to the space like:

--post-data '1=1'\ http://domain1.com/page.html

is making it be treated as one url, but the -- appear to be stripped from the --post-data argument.

  • 3
    Try -L 1 instead of -n 1. This will run wget once for each line of input rather than once for each word of input. – Mark Plotnick Jan 23 '14 at 0:09
  • Thank you Mark Plotnick, that has done the trick. Feel free to post the answer if you would like the rep. from this post. – Kohjah Breese Jan 23 '14 at 0:12
3

To make xargs run your command once for each line of input, give it the -L 1 option (and remove the -n 1 option, since they are mutually exclusive). The standards document for xargs says this:

-L number
The utility shall be executed for each non-empty number lines of arguments
from standard input. A line is considered to end with the first <newline>
unless the last character of the line is a <blank>; a trailing <blank>
signals continuation to the next non-empty line, inclusive.
1

Open the terminal and run:

cat urls.txt | sed "s/[\"\<\>' \t\(\);]/\n/g" | grep "http://" | sort -u | xargs -n 1 -P 10 wget -q
0

You don't need xargs,you can do this with a simple bash loop Either keep your URLs as a simple list and add the options directly to wget:

while IFS= read -r url; do wget --post-data '1=1' "$url"; done < urls.txt

or include them in the file and pass them to wget:

while IFS= read -r url; do wget "$url"; done < urls.txt

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.