Redirecting pv output to file

Question

I'm using the pv utility to asses the speed of log entries being added in a log file, like this:

tail -f -n 0 mylog.log | pv -lr -t -i 5 > /dev/null

This outputs the addition speed and refreshes every 5 sec:

0:00:05 [  10/s]

Now, I would like to redirect the current value ("10") to a file and refresh it every 5s. Not append it, just overwrite the previous value in the file. Any idea on how to do the redirection to file? I'm pretty new in unix and haven't got it to work yet.

Thanks!

Answer 1

pv is writing its diagnostics to stderr, so you can start with:

tail -f -n 0 mylog.log | pv -lr -t -i 5 > /dev/null 2> output-file

This will merely append, so output-file will end up with all of the data with lines terminated by carriage returns without linefeeds. (I don't have access to pv at the moment, so this is all conjecture.) To get the data you want, try some simple post processing along the lines of:

tail -f mylog.log | pv -lr -t -i 5 2>&1 > /dev/null | 
  tr /\\r \ \\n | tr -d [] | while read stamp value;
  do echo $value > output-file; done

The shell technique above allows you to direct stderr into the pipe for post-processing. Using tail like you're doing is a little odd, and pv is overkill for this. If you have such a long delay (5 seconds is a long time), it's probably acceptable to just use wc on each iteration and do subtraction:

echo 0 > output-file
while sleep 5; do
    expr $(wc -l < mylog.log) - $(cat output-file) > output-file
done

Note that this prints the difference over 5 seconds rather than a rate per second. expr does not handle non-integer arithmetic very well, so dividing by 5 is problematic. You could use bc or dc to get more precision, but you're probably better off not even doing subtraction and just writing the total number of lines to the file every few seconds.

Answer 2

As I can't add a comment, I want to follow up on this one

Use the -f flag on pv to force its output to a file.

-f, --force Force output. Normally, pv will not output any visual display if standard error is not a terminal. This option forces it to do so.

e.g. tail -f -n 0 mylog.log | pv -lrf -t -i 5 > /dev/null 2> output-file

Answer 3

If the byte count of the file is OK instead of the line count, you can do this:

while true; do
  ls -s --block-size=1 mylog.log > output-file
  sleep 5
end

This is much more efficient than using PV. The "pv" command is burning a lot of CPU by copying all the bytes in your log.

Answer 4

The other answers tell how to get pv to save to a file, but some are incomplete. pv terminates each logged line with a \r (carriage return or CR) and not a \n (newline). If you, for example, cat pv-output-file, it won't print each line separately to the screen, rather it'll print each line and then overwrite it almost instantly. more seems to behave somewhat better, but still not great, and less will give you the whole pv log on a single line rendered with a bunch of ^M's at what should be the end of each line.

To get it to render well in a text viewer, you'll still need to do something to translate the CR's to newlines. There are any number of ways to do this in unix including tr, sed, awk, and even one of the dos2unix variants (I think it's mac2unix or dos2unix -c mac).

If you have perl installed, the following one-liner will print each line of pv-output-file on a separate line.

perl -pe 's/\r/\n/g' pv-output-file

You can redirect the output of that into a text viewer like less or more to scroll through the lines. If you'd just want to 'fix' it in place, do perl -pi -e [...] as above.