4

I'm running a command:

cat urls.txt | xargs -L 1 -P 5 timeout 40 wget

This downloads all the URLs in urls.txt

This used to work fine, but recently I changed the URLs I am downloading from something like: domain.com/54f-5g4/ to something like domain.com/this-that/

And for some reason I am now getting:

cat: write error: Broken pipe

I've looked at the commands being executed and they all look fine, i.e. no unescaped characters.

Does anyone have any idea what the problem may be? Or how to get more debugging information? I believe the problem is with commands after the pipe.

Update: I found the error. It looks like I accidently typed a ' in my urls.txt file. If you are getting this error, you have a problem with your command after the pipe.

2
  • 2
    Correct me if I'm wrong.. If your purpose is to download whatever specified on that urls.txt file; well you can simply use wget -i urls.txt :)
    – AzkerM
    Feb 16, 2014 at 16:35
  • wget only allows one connection at a time, using xargs, you can have multiple connections running simultaneously. Feb 26, 2014 at 1:33

1 Answer 1

8

When you use a pipe in a shell:

command_1 | command_2

the output of command_1 is piped to command_2 as input. If command_2 ends before reading all the output of command_1 (e.g. in your case because it found something wrong as you said in your update that you accidentally typed a '), then command_1 may complain that its output pipe was prematurely closed.

1
  • 1
    Just a small follow up, why this happens on some systems and doesn't on others. I mean the message e.g. is not shown on my bash, but is shown on my server. Feb 11, 2021 at 12:47

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .