# cat and Pipe to Run Command Won't Take Two Arguments

I have been using this command:

cat urls.txt | xargs -n 1 -P 10 wget -q


to download text files of URLs. This worked fine when my URL file was like:

http://domain1.com/page.html
http://domain2.com/page.html
http://domain3.com/page.html


However I now need to download text files of URLs and post data, like:

--post-data '1=1' http://domain1.com/page.html
--post-data '1=1' http://domain2.com/page.html
--post-data '1=1' http://domain3.com/page.html


When using the above cat command it tries to download the URL and then the post data as a URL. e.g. in the above example, it would download http://domain1.com/page.html and then try and download --post-data 1=1, then http://domain2.com/page.html and so on.

Is there anyway to get cat to send each line of the URLs files only?

Update: I've found that by adding an escape to the space like:

--post-data '1=1'\ http://domain1.com/page.html


is making it be treated as one url, but the -- appear to be stripped from the --post-data argument.

• Try -L 1 instead of -n 1. This will run wget once for each line of input rather than once for each word of input. – Mark Plotnick Jan 23 '14 at 0:09
• Thank you Mark Plotnick, that has done the trick. Feel free to post the answer if you would like the rep. from this post. – Kohjah Breese Jan 23 '14 at 0:12

To make xargs run your command once for each line of input, give it the -L 1 option (and remove the -n 1 option, since they are mutually exclusive). The standards document for xargs says this:

-L number
The utility shall be executed for each non-empty number lines of arguments
from standard input. A line is considered to end with the first <newline>
unless the last character of the line is a <blank>; a trailing <blank>
signals continuation to the next non-empty line, inclusive.


Open the terminal and run:

cat urls.txt | sed "s/[\"\<\>' \t;]/\n/g" | grep "http://" | sort -u | xargs -n 1 -P 10 wget -q


You don't need xargs,you can do this with a simple bash loop Either keep your URLs as a simple list and add the options directly to wget:

while IFS= read -r url; do wget --post-data '1=1' "$url"; done < urls.txt  or include them in the file and pass them to wget: while IFS= read -r url; do wget "$url"; done < urls.txt