bash standard output can not be redirected into file

Question

I am reading 'advanced bash script', in Chapter 31, there is a problem. I can not figure it out.

tail -f /var/log/msg | grep 'error' >> logfile

Why is there nothing to output into logfile?
can you offer me an explanation? thank you in advance

Answer 1

As @chepner comments, grep is using a larger buffer (perhaps 4k or more) to buffer its stdout. Most of the standard utilities do this when piping or redirecting to a file. They typically only switch to line-buffered mode when outputting directly to the terminal.

You can use the stdbuf utility to force grep to do line buffering of its output:

tail -f /var/log/msg | stdbuf -oL grep 'error' >> logfile

As an easily observable demonstration of this effect, you can try the following two commands:

for ((i=0;;i++)); do echo $i; sleep 0.001; done | grep . | cat

and

for ((i=0;;i++)); do echo $i; sleep 0.001; done | stdbuf -oL grep . | cat

In the first command, the output from grep . (i.e. match all lines) be buffered going into the pipe to cat. On mine the buffer appears to be about 4k. You will see the ascending numbers output in chunks as the buffer gets filled and then flushed.

In the second command, grep's output into the pipe to cat is line-buffered, so you should see terminal output for every line, i.e. more-or-less continuous output.