How does I/O occur for large data files?

Question

Assume that we have a 4GB RAM processor. Let's say we want to play a 4GB movie in VLC player. Now, processor first starts VLC Process and then starts to load 4Gb data. My question is how does this data transfer occur, Is it like getting almost all of 4Gb and storing some in swap area and bringing whenever it is required or first getting the starting data and continuosly stream data between RAM and hard disk drive?

Answer 1

At the level a user-mode process, there are two basic ways in which sequential file IO like playing a movie from start to finish can take place:

• The process can allocate a buffer, which is typically sized somewhere between a few kilobytes up to a megabyte or so, but rarely any larger than that, and then asks the operating system to fill it up. When it is finished with the data from that buffer, it then asks the operating system for more, and so on, until it reaches the end of the file. As a slight variant on this, in order to reduce latency it may allocate two buffers, and ask the operating system to fill the second one while it is processing the data from the first, and then swap between the buffers when it finishes processing the data from the first.

• Or, it might instead ask the operating system to "map" the file (or a section of it) into memory. In this case, the operating system tells the processor that the memory is currently not available, and then when the reading process tries to use the memory, the processor notices this and switches back to the operating system, causing the operating system to load in the next page of data from the disk. This may be very slightly more efficient because the operating system is able to manage the memory itself rather than having to allocate its own internal buffers and then copy the data after it has been read into the process's buffers, but the effects are likely very difficult to notice in most cases.

My suspicion is that most media players use the first approach, probably with the variant using two buffers. The third can be more efficient, but it has a problem: you need to allocate addresses for the entire content of the file[1], which can be a problem with large files under 32-bit operating systems, so any media player designed to work under such a system is unlikely to do this.

At the operating system level, the kernel sees a series of requests for sections of files to be loaded, and the filesystem driver translates those to requests for disk blocks to be loaded. The IO manager allocates memory blocks from its buffer pool (aka IO cache) and asks the disk device driver to transfer the disk blocks' contents into the memory blocks. Once this is done, either the data is copied into the user process buffer (case 1 above) or the system buffer page is mapped into the process's memory space using a page table and the process resumes (case 2).

A modern operating system is also likely to notice that the process is reading blocks sequentially and start prefetching them before requests come in to improve performance.

[1] - you can just map part of the file and then move the window around, but my understanding is that very few programs actually do this.