“tail -f” makes disk full?

Question

our application server (sunOS) always gets disk full. and our Infrastructure team said it's caused by too many "tail -f" processes. Because the app rotates log file frequently, it caused the dead link and no disk space? I've never heard of this before. does that command really cause disk full?

Answer 1

The space a file occupies cannot be reclaimed until all references to that file are gone. Therefore, any process that has the file open will prevent the file from being deleted from the disk.

An active tail -f following the file, for example.

If these files need to be deleted to free disk space (e.g. because they are very big, or there are very many of them), then having processes lying around that hold references to them will prevent their deletion, and eventually lead to the disk filling up.

Edit in response to the comment on the other answer:

The diagnostics you report are exactly what you would expect to see in the situation that Adam and I describe. df reports that 56G of disk are in use, and du reports that only 10G are visible in the folder. The discrepancy is because there are 46G worth of files that have been removed from the folder, but cannot be physically removed from disk because some processes are holding a references to them.

It's easy enough to experiment with this yourself: find a filesystem it's safe to play with, and create a humongous file. Write a C program that opens the file and goes into an infinite loop. Now, do the following:

• Start the program
• Check the output of df
• rm the file
• Check the output of df again
• Stop your program
• Check the output of df again

You will see that the output of df doesn't change after rming the file, but does change once you stop the program (thus removing the last reference to the file).

If you need even more evidence that this is what's going on, you may be able to get information from the /proc filesystem, if you have it. Specifically, find the PID of one of the tail -f processes (or other processes you think might be the cause), and look at the directory /proc/<pid>/fd to see all of the files it has open.

(I don't have *nix at home, so I can't actually check to see just what you'll see /proc/<pid>/fd in this situation)

Answer 2

tail is a command to view the end of a file, and -f performs that in real-time, updating the display whenever the file itself is modified. It allows real-time views of log files.

tail can cause problems in two ways:

1. If tail -f is being misused to write to a file instead of an interactive console, it is an inefficient means of copying a file, and it's creating duplicate logs.
2. tail -f keeps the log file alive, so maintenance tasks trying to automatically delete log files fail. This throws off the log file rotation by not allowing the old one to age off.

If your infrastructure team is complaining about access to files less than a week old, you really need more drive space and/or a less-verbose logging strategy, since they can't keep enough logs alive in case something goes wrong and needs to be tracked. If the logs are older than that, they may have a good point, and excessive usage of tail- like anything else keeping the file open- may be preventing it from being deleted in a timely manner.

The command on its own is unlikely to be filling the disk, but it may be preventing disk hygiene operations.