How are files written? Why do I not see my data written immediately?

Question

I understand the general process of writing and reading from a file, but I was curious as to what is happening under the hood during file writing. For instance, I have written a program that writes a series of numbers, line by line, to a .txt file. One thing that bothers me however is that I don't see the information written until after my c program is finished running. Is there a way to see the information written while the program is running rather than after? Is this even possible to do? This is a hard question to phrase in one line, so please forgive me if it's already been answered elsewhere.

The reason I ask this is because I'm writing to a file and was hoping that I could scan the file for the highest and lowest values (the program would optimally be able to run for hours).

Answer 1

Research buffering and caching.

There are a number of layers of optimisation performed by:

1. your application,
2. your OS, and
3. your disk driver,

in order to extend the life of your disk and increase performance.

With the careful use of flushing commands, you can generally make things happen "quite quickly" when you really need them to, though you should generally do so sparingly.

Flushing can be particularly useful when debugging.

The GNU C Library documentation has a good page on the subject of file flushing, listing functions such as fflush which may do what you want.

Answer 2

You observe an effect solely caused by the C standard I/O (stdio) buffers. I claim that any OS or disk driver buffering has nothing to do with it.

In stdio, I/O happens in one of three modes:

1. Fully buffered, data is written once BUFSIZ (from <stdio.h>) characters were accumulated. This is the default when I/0 is redirected to a file or pipe. This is what you observe. Typically BUFSIZ is anywhere from 1k to several kBytes.
2. Line buffered, data is written once a newline is seen (or BUFSIZ is reached). This is the default when i/o is to a terminal.
3. Unbuffered, data is written immediately.

You can use the setvbuf() (<stdio.h>) function to change the default, using the _IOFBF, _IOLBF or _IONBF macros, respectively. See your friendly setvbuf man page.

In your case, you can set your output stream (stdout or the FILE * returned by fopen) to line buffered.

Alternatively, you can call fflush() on the output stream whenever you want I/O to happen, regardless of buffering.

Answer 3

Indeed, there are several layers between the writing commands resp. functions and the actual file.

First, you open the file for writing. This causes the file to be either created or emptied. If you write then, the write doesn't actually occur immediately, but the data are cached until the buffer is full or the file is flushed or closed.

You can call fflush() for writing each portion of data, or you can actually wait until the file is closed.

Answer 4

Yes, it is possible to see whats written in the file(s). If you programm under Linux you can open a new Terminal and watch the progress with for example "less Filename".