40

I'm tailing a log file with -f flag. Then I'm piping this to grep, to find only lines that contain "X". That's working perfectly fine. Now I want to pipe this again into another grep, that will remove all the lines containing "Y". When I add the second pipe, the file stop refreshing and it looks like no data is coming.

This is the command that works: tail -f my_file.log | grep "X"

This is the command that doesn't: tail -f my_file.log | grep "X" | grep -v "Y"

How should I structure this so that the command works?

2
54

As the output of grep is buffered, use --line-buffered option of grep to enable line buffering:

tail -f /path/to/log | grep --line-buffered 'X' | grep -v 'Y'

If your grep does not have the option, you can use stdbuf as an alternative:

tail -f /path/to/log | stdbuf -oL grep 'X' | grep -v 'Y'
4
  • 1
    I am wondering how stdbuf tells libstdbuf.so which settings to use. – kasperd Oct 14 '15 at 7:21
  • @kasperd: Environment variables. – Nate Eldredge Oct 14 '15 at 18:47
  • 1
    @NateEldredge I already looked for environment variables in the output from diff -u <(env) <(stdbuf env) and found none. But now I realize that what I should have tested was diff -u <(env) <(stdbuf -oL env). – kasperd Oct 14 '15 at 19:59
  • I am also same sort of problem. my case is, I need to print all lines contain 'aaa' and 'bbb'. Above first solution not working. Second solution is working for 'aaa' exist and 'bbb' not exist. both exist not working. giving no output. my command looks like this: tail -f test.txt | stdbuf -oL grep 'aaa' | grep 'bbb' it not giving any output. Could you please help me. – Sun Apr 18 '17 at 4:54
19

I normally find more useful awk for these kind of logical checks:

tail -f /path/to/log | awk '/X/ && !/Y/'
#                           ^^^    ^^^^
#                   this I want    but not this

Tested by having two tabs, one in which I keep writing seq 20 >> myfile and the other one having for example tail -f myfile | awk '/3/ && !/13/'.

15

Another approach would be to use a single grep invocation instead of two and so avoid the buffering issue. Just use a regular expressions that matches lines consisting of 0 or more non-Y characters, then an X and then 0 or more non-Ys again till the end of the line"

tail -f /path/to/log | grep '^[^Y]*X[^Y]*$'

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.