4

I am running Ubuntu 14.04.3, it's uptodate. I don't know why, for a few days I began to take grep: write error: Broken pipe message on launching gnome-terminal . It seems to be harmless but it bothers me. How can I debug it?

EDIT: I moved aliases and functions each to separate files such as .bash_aliases and .bash_functions and added a command to load them from .bashrc

 if [ -f ~/.bash_aliases ]; then
. ~/.bash_aliases
 fi

 if [ -f ~/.bash_functions ]; then
. ~/.bash_functions
 fi

If I don't load .bash_functions problem disappears.

I am trying to find the faulty one by disabling each function one by one.

This one gives me the same error but when I disable it I keep getting the same error, so I may have more faulty functions.

 ls -lt  $PWD| grep ^d | head -1 | cut -b 51- 

 grep:  development
 write error: Broken pipe

I wonder why I begin to take that error.

EDIT2:

I found a similar problem here boken pipe

The root of the problem also seems similar.

I tried the given test command in the link which have the same error:

 bash -c '(while echo foo; do :; done); echo status=$? >&2' |  head
 foo
 foo
 foo
 foo
 foo
 foo
 foo
 foo
 foo
 foo
 bash: line 0: echo: write error: Broken pipe
 status=0

EDIT3:

Though that unbuffer workaround I posted below as an answer to my own question works, I am not satisfied with it, but my knowledge about debugging is limited. Acoording to this link https://lists.gnu.org/archive/html/bug-coreutils/2007-11/msg00080.html it stems from SIGPIPE trap by another task, and this link https://lists.gnu.org/archive/html/bug-coreutils/2007-11/msg00154.html pinpoints the exact cause of the problem, it's one of the pam authentication module which I am in trouble with it recently.

6
  • 1
    Look for grep in your .bashrc, .profile, etc. And is it only in gnome-terminal? Not in xterm, or in the TTYs?
    – muru
    Feb 14, 2016 at 12:32
  • @muru I already checked .bashrc and .profile I haven't added any new functions recently. I have a number of grep and | (pipe) in functions in them but they are there for months.
    – kenn
    Feb 14, 2016 at 15:08
  • @muru when I run your command shopt -s nullglob; ls -t */ | head -1 I get development/: ls: write error: Broken pipe output
    – kenn
    Feb 14, 2016 at 18:47
  • @muru I posted output of ls -dt */ in development directory. Added it to my question.
    – kenn
    Feb 14, 2016 at 19:10
  • I meant, I wanted the output of printf '%q' */ in whichever directory you were originally in, and that I meant the replacement function would be: shopt -s nullglob; ls -dt */ | head -1. Sorry for the miscommunication.
    – muru
    Feb 14, 2016 at 19:11

2 Answers 2

5

There's a great explanation of this problem on this Super User answer: How can I fix a Broken Pipe error?.

Commands in pipes are run asynchronously: this means that in a pipe such as command1 | command2 there's no guarantee that command1 will end before command2.

When using [...] | grep | head -n 1, head ends as soon as it has read one line; if this happens before grep has finished writing to the pipe, grep receives a SIGPIPE signal and errors out.

As explained in the answer below that Super User answer, a workaround is to pipe the output of what's before head in the pipeline to tail -n +1 first, which will ignore the SIGPIPE signal:

command | tail -n +1 | head -n 1

But in this case there's not even any need for head, since grep has an option to print only the first match:

[...] | grep -m 1
16
  • Thank you for responding. How can I apply it to my particular case for ls -lt $PWD| grep ^d | head -1 | cut -b 51- and generalize it?
    – kenn
    Feb 14, 2016 at 19:19
  • @kenn I had a misconception about pipes and the answer wasn't technically accurate before, it should be now. Also see the update, there's no real need for grep | head.
    – kos
    Feb 14, 2016 at 19:57
  • @kenn Meaning: ls -lt $PWD| grep -m 1 ^d | cut -b 51-.
    – kos
    Feb 14, 2016 at 20:18
  • ls -lt $PWD| grep -m 1 ^d | cut -b 51- still gives me the same error development ls: write error: Broken pipe :(
    – kenn
    Feb 14, 2016 at 20:24
  • @kenn Looks like the same is happening with cut as well. What about ls -lt $PWD| grep ^d | tail -n +1 | head -n 1 | cut -b 51-?
    – kos
    Feb 14, 2016 at 20:27
3

After hours of struggling with the problem I found a working workaround (I hope so)

Problem seems to be deeper and complicated. Many people met the same bug. Fixing it is beyond my coverage.

Closest workaround posted here how-can-i-fix-a-broken-pipe-error by Andrew Beals at bottom like :

ls -lt $PWD|dd obs=1M | grep -m 1 ^d | cut -b 51-

is not neat.

When I intuited that it's related to pipe buffer I gave a shot to unbuffer command like :

 unbuffer ls -lt $PWD| grep -m 1 ^d | cut -b 51-

It works well.

I hope somebody posts the real cause of the problem.

EDIT: A bash Guru would suggest this simple solution , redirecting stderr to /dev/null

 ls -lt $PWD 2>/dev/null | grep -m 1 ^d | cut -b 51-
3
  • Killing stderr's output hides other error conditions from you. When working with large output today, I found that I had to increase the buffer size in dd again and though about the problem a little more. In the strictest sense, the pipe is "broken", but the shell shoudn't be telling the user about it and for the simple case of excess data, but not too much excess data as it's all sitting in buffers, SIGPIPE does not happen: cat /etc/passwd | grep -m1 ssh For "Big Data" problems here such as your example, I would change it to ls -lt $PWD | (grep -m 1 ^d ; dd of=/dev/null) | cut -b 51- May 27, 2016 at 18:57
  • Yes, this takes longer, but avoids having to do janky things such as redirecting stderr to /dev/null in the script. However, it's a bad solution in and of itself, as it's preventing the upstream command from exiting after you're done consuming its data. More discussion here: pixelbeat.org/programming/sigpipe_handling.html May 27, 2016 at 19:02
  • @AndrewBeals Thank you for further info and suggestion. Beside the point do you think Gnome developers are programmers or self-satisfying obfuscators?
    – kenn
    May 27, 2016 at 20:37

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .