4

I'm running a command:

cat urls.txt | xargs -L 1 -P 5 timeout 40 wget

This downloads all the URLs in urls.txt

This used to work fine, but recently I changed the URLs I am downloading from something like: domain.com/54f-5g4/ to something like domain.com/this-that/

And for some reason I am now getting:

cat: write error: Broken pipe

I've looked at the commands being executed and they all look fine, i.e. no unescaped characters.

Does anyone have any idea what the problem may be? Or how to get more debugging information? I believe the problem is with commands after the pipe.

Update: I found the error. It looks like I accidently typed a ' in my urls.txt file. If you are getting this error, you have a problem with your command after the pipe.

  • 2
    Correct me if I'm wrong.. If your purpose is to download whatever specified on that urls.txt file; well you can simply use wget -i urls.txt :) – AzkerM Feb 16 '14 at 16:35
  • wget only allows one connection at a time, using xargs, you can have multiple connections running simultaneously. – Kohjah Breese Feb 26 '14 at 1:33
8

When you use a pipe in a shell:

command_1 | command_2

the output of command_1 is piped to command_2 as input. If command_2 ends before reading all the output of command_1 (e.g. in your case because it found something wrong as you said in your update that you accidentally typed a '), then command_1 may complain that its output pipe was prematurely closed.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.