Ask Ubuntu is a question and answer site for Ubuntu users and developers. It's 100% free.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have been using this command:

cat urls.txt | xargs -n 1 -P 10 wget -q

to download text files of URLs. This worked fine when my URL file was like:

However I now need to download text files of URLs and post data, like:

--post-data '1=1'
--post-data '1=1'
--post-data '1=1'

When using the above cat command it tries to download the URL and then the post data as a URL. e.g. in the above example, it would download and then try and download --post-data 1=1, then and so on.

Is there anyway to get cat to send each line of the URLs files only?

Update: I've found that by adding an escape to the space like:

--post-data '1=1'\

is making it be treated as one url, but the -- appear to be stripped from the --post-data argument.

share|improve this question
Try -L 1 instead of -n 1. This will run wget once for each line of input rather than once for each word of input. – Mark Plotnick Jan 23 '14 at 0:09
Thank you Mark Plotnick, that has done the trick. Feel free to post the answer if you would like the rep. from this post. – Kohjah Breese Jan 23 '14 at 0:12
up vote 3 down vote accepted

To make xargs run your command once for each line of input, give it the -L 1 option (and remove the -n 1 option, since they are mutually exclusive). The standards document for xargs says this:

-L number
The utility shall be executed for each non-empty number lines of arguments
from standard input. A line is considered to end with the first <newline>
unless the last character of the line is a <blank>; a trailing <blank>
signals continuation to the next non-empty line, inclusive.
share|improve this answer

Open the terminal and run:

cat urls.txt | sed "s/[\"\<\>' \t\(\);]/\n/g" | grep "http://" | sort -u | xargs -n 1 -P 10 wget -q
share|improve this answer

You don't need xargs,you can do this with a simple bash loop Either keep your URLs as a simple list and add the options directly to wget:

while IFS= read -r url; do wget --post-data '1=1' "$url"; done < urls.txt

or include them in the file and pass them to wget:

while IFS= read -r url; do wget "$url"; done < urls.txt
share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.