need help with grep on output of another grep command

Question

I have a file test.log. Very big log file. It has different levels of logging. For eg, trace, debug, info, warning and error.

Obviously trace level messages are just spamming at high speeds. I wanted to see all the messages without trace level logs.

So I did this:

cat test.log | grep -v "trace"

Works good.

Now I want to filter the remaining messages based on a certain keyword keyword1.

So I did this:

cat test.log | grep -v "trace" | grep "keyword1"

Works good.

Now I want to get that same output on a continual basis, I thought of replacing cat with tail -f.

tail -f test.log | grep -v "trace" | grep "keyword1"

But this does not work. I get no output at all.

What am I doing wrong? And how can I get my desired filtered 'tail & follow' output.

Thanks for the help.

(btw, I am using cygwin... if that matters in any way)

Answer 1

You are encountering buffering issues: for performance reasons grep will keep quite a bit of output in its buffer and will output the whole chunk in one go. That means that when a line is read from the input, grep will send it to stdout once it has read (and let through) even more lines - which can be quite some time later when dealing with a log file that is being read-in using tail -f.

Many grep variants have a switch to turn-on line-buffered mode, which will output each line on its own - with some performance loss. For example GNU grep has a --line-buffered option to achieve this effect.

Just add that option (or the appropriate one for your version of grep) to all your grep invocations and you see some output as soon as a matching like is added to the log file.

Answer 2

Your code is working fine, you're just encountering buffering delays. So you will see a long period of nothing, followed by a short burst of lots of text, followed by another wait. Read http://perl.plover.com/FAQs/Buffering.html for an explanation of what is going on.

Answer 3

tail -f follows the "new incoming lines" of a file. The periodic output will never reach the piped grep commands (at least not, until tail is terminated).

To periodically "follow" changes in those log files, you might want to use watch instead:

watch -n 1 -- 'tail -n 20 test.log | grep -v trace | grep keyword1'

This'll update every second (-n 1) the last 20 lines of the log file.