51

I'm tailing a log file with -f flag. Then I'm piping this to grep, to find only lines that contain "X". That's working perfectly fine. Now I want to pipe this again into another grep, that will remove all the lines containing "Y". When I add the second pipe, the file stop refreshing and it looks like no data is coming.

This is the command that works: tail -f my_file.log | grep "X"

This is the command that doesn't: tail -f my_file.log | grep "X" | grep -v "Y"

How should I structure this so that the command works?

2

3 Answers 3

62

As the output of grep is buffered, use --line-buffered option of grep to enable line buffering:

tail -f /path/to/log | grep --line-buffered 'X' | grep -v 'Y'

If your grep does not have the option, you can use stdbuf as an alternative:

tail -f /path/to/log | stdbuf -oL grep 'X' | grep -v 'Y'
4
  • 1
    I am wondering how stdbuf tells libstdbuf.so which settings to use.
    – kasperd
    Oct 14, 2015 at 7:21
  • @kasperd: Environment variables. Oct 14, 2015 at 18:47
  • 1
    @NateEldredge I already looked for environment variables in the output from diff -u <(env) <(stdbuf env) and found none. But now I realize that what I should have tested was diff -u <(env) <(stdbuf -oL env).
    – kasperd
    Oct 14, 2015 at 19:59
  • I am also same sort of problem. my case is, I need to print all lines contain 'aaa' and 'bbb'. Above first solution not working. Second solution is working for 'aaa' exist and 'bbb' not exist. both exist not working. giving no output. my command looks like this: tail -f test.txt | stdbuf -oL grep 'aaa' | grep 'bbb' it not giving any output. Could you please help me.
    – Sun
    Apr 18, 2017 at 4:54
22

I normally find more useful awk for these kind of logical checks:

tail -f /path/to/log | awk '/X/ && !/Y/'
#                           ^^^    ^^^^
#                   this I want    but not this

Tested by having two tabs, one in which I keep writing seq 20 >> myfile and the other one having for example tail -f myfile | awk '/3/ && !/13/'.

15

Another approach would be to use a single grep invocation instead of two and so avoid the buffering issue. Just use a regular expressions that matches lines consisting of 0 or more non-Y characters, then an X and then 0 or more non-Ys again till the end of the line"

tail -f /path/to/log | grep '^[^Y]*X[^Y]*$'

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .