2

This question helped me understand how to combine stderr and stdout:

How to redirect stderr to a file

with this command:

gunzip -vc /opt/minecraft/wonders/logs/20* 2>&1

But how may one insert a line break between the two so that the output can appear on separate lines instead of clumped together?

The stderr comes first, which is always 1 line, something like this:

/opt/minecraft/wonders/logs/2017-08-28-2.log.gz:

Whereas the stdout is usually (but not always) multiple lines, like:

06:17:05: Starting minecraft server version 1.10.2
06:18:21: Loading properties
06:18:21: Default game type: SURVIVAL
06:18:21: Generating keypair
06:18:21: Starting Minecraft server on *:25565
06:18:22: Using default channel type

But because there is no newline between them the first line from each file to be unzipped is always something like this:

/opt/minecraft/wonders/logs/2018-08-18-2.log.gz:    06:17:05: Starting minecraft server version 1.10.2

I need these two parts on different lines so I can filter them with grep. But I also want the corresponding stdout directly following its stderr (its filename) so I know which file it comes from. Like this:

/opt/minecraft/wonders/logs/2018-08-18-2.log.gz:    
06:17:05: Starting minecraft server version 1.10.2
06:18:21: Loading properties
06:18:21: Default game type: SURVIVAL
06:18:21: Generating keypair
06:18:21: Starting Minecraft server on *:25565
06:18:22: Using default channel type

This is the command I am piping into using with grep:

egrep -iv "^\[.*(stopping|starting|saving|keep|spawn|joined|lost|left|: (<.*>|\/)|\/(help|forge))

Its purpose is to remove certain lines from the log files, leaving the pertinent ones.

  • 1
    Can you add more detail as to how exactly you're trying to filter with grep ? Especially, can you explain the end goal you're trying to achieve ? If you need filtering on both streams, you could probably do gunzip -vc /opt/minecraft/wonders/logs/20* > >(grep 'stdin pattern' ) 2> >(grep 'stderrpattern' ) – Sergiy Kolodyazhnyy Aug 20 '18 at 22:34
2

Here is one simple way to do this, assume this command:

ls -d Documents/ NotExist/ /dev/null

the output is:

ls: cannot access 'NotExist/': No such file or directory
/dev/null  Documents/

the first line is stderr and the second is stdout, if I pipe the output to xargs I can print a new line after stderr and then print the stdout:

ls -d Documents/ NotExist/ /dev/null | xargs echo -e '\n'

which gives me:

ls: cannot access 'NotExist/': No such file or directory

 /dev/null Documents/
  • This didn't work. I added more detail to my question which could explain why. The error message I get is xargs: unmatched single quote; by default quotes are special to xargs unless you use the -0 option. Using the -O option, double quotes or no quotes removed the error message but now only a small bit of output shows up, as far as I can tell the first line of each file from stdout. Thanks. – Jesse Yishai Aug 20 '18 at 22:30
2

It's possible with duplicating file descriptors. Long story, short: pipe stderr, where it can be processed, and send stdin to the controlling terminal ( short hand for that is /dev/tty ) so that it still shows on the screen:

$ stat /etc/passwd noexist 2>&1 > /dev/tty | while IFS= read -r line; do echo "---"; echo "$line"; echo "---"; done
  File: /etc/passwd
  Size: 2380        Blocks: 8          IO Block: 4096   regular file
Device: 801h/2049d  Inode: 937653      Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2018-08-20 13:05:01.171247259 +0800
Modify: 2018-07-24 04:51:36.799057737 +0800
Change: 2018-07-24 04:51:36.867020035 +0800
 Birth: -
---
stat: cannot stat 'noexist': No such file or directory
---

Here I'm separating stderr with --- line, but you get the idea.

Another way, if you want to visually separate them is to colorize the output. Based on Balazs Pozar's answer, you could do:

command 2> >(while read line; do echo -e "\e[01;31m$line\e[0m" >&2; done)

enter image description here


To address the edit on the question, if you need to filter both stdout and stderr with grep (for example, if you're looking for a pattern in stdout, but another pattern in stderr ), you use multiple process substitutions >():

gunzip -vc /opt/minecraft/wonders/logs/20* > >(grep 'stdin pattern' ) 2> >(grep 'stderrpattern' )

Addressing the comments, if you only need to separate the filename /opt/minecraft/wonders/logs/2017-08-28-2.log.gz: from further content, you could use sed:

gunzip -vc /opt/minecraft/wonders/logs/20* | 
sed -r 's/^(\/opt\/minecraft\/wonders\/logs\/2018-08-18-2.log.gz:) *(.*)$/\1\n\2/'

Test:

$ echo /opt/minecraft/wonders/logs/2018-08-18-2.log.gz:    06:17:05: Starting minecraft server version 1.10.2 | sed -r 's/^(\/opt\/minecraft\/wonders\/logs\/2018-08-18-2.log.gz:) *(.*)$/\1\n\2/'
/opt/minecraft/wonders/logs/2018-08-18-2.log.gz:
06:17:05: Starting minecraft server version 1.10.2
  • 1
    @JesseYishai I think there's no way to reorder them, or no simple method at least, without writing whole program that monitors both streams and maybe timestamps them. – Sergiy Kolodyazhnyy Aug 20 '18 at 22:44
  • All 3 methods did indeed separate the stdout and the stderr, but also produced the results without proper order, by which I mean the stdout of a file is not always lined up (following) its stderr. They are out of order. Also, I don’t need filtering on the stderr, except for the option to omit it entirely if it is followed by certain elements (or lack of them) in the stdout. Thanks. – Jesse Yishai Aug 20 '18 at 22:46
  • @JesseYishai So you're basically trying to do grep -v 'pattern' on stderr - hide "pattern" but show other errors ? – Sergiy Kolodyazhnyy Aug 20 '18 at 22:50
  • I am grep -v 'pattern' on stdout, hiding certain elements, with the heading from stderr (the filename the stdout is coming from) preceding it. – Jesse Yishai Aug 20 '18 at 22:52
  • 1
    You might want to use a different sed delimiter, like |, so you don't have to escape the slashes: 's|^(/opt/minecraft/wonders/logs/2018-08-18-2.log.gz:) *(.*)$|\1\n\2|' – wjandrea Aug 21 '18 at 0:14
1

What's really happening I think is that stdout is getting mixed into stderr, because the former is line-buffered and the latter unbuffered by default. To illustrate:

$ gunzip -vc ../foo.txt.gz
../foo.txt.gz:  some text here
more text here
 26.7%

(note that -v is causing two pieces of information to be emitted on stderr: the filename, and the % compression - with standard output falling in between).

If you line-buffer stderr as well, the streams will appear on separate lines:

$ stdbuf -eL gunzip -vc ../foo.txt.gz
some text here
more text here
../foo.txt.gz:   26.7%

however not necessarily in the order that you require - you might try using the sponge utility to soak up all of stdout before outputing it, so that the stderr output (filename and %) appears in the terminal first:

gunzip -vc ../foo.txt.gz | sponge
../foo.txt.gz:   26.7%
some text here
more text here

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.