How Database Formats Files for Disk

Question

This SO answer outlines how MySQL organizes its database into files, but doesn't describe the structure of the files themselves.

I am wondering, at the physical level, how relational database management systems store their records in files. What the structure of the file is roughly, where the columns/values are (if it's just a CSV), and how it is traversed.

Answer 1

I think it is opportune to start pointing out that, in a relational database —as per Dr. E.F. Codd— information is represented by means of one and only one structure, the relation, which is usually portrayed as a table.

The relational model deals exclusively with aspects of the logical level of abstraction, so a table (an abstraction) is part of the logical level of a database (it is susceptible of undergoing logical operations, e.g., projections, joins, restrictions, etc.), so it is completely independent from the physical storage utlized by the database management system (DBMS for brevity) of use.

That being so, since the relational model has to do solely with logical-level aspects, it offers physical data independence, which sets DBMS (e.g., SQL platforms like IBM DB2, Microsoft SQL Server, PostgreSQL, etc. —that although quite powerful, strictly speaking, are semi-relational, i.e., they do not comply fully with the stipulations of the relational framework—) free to (a) store at the physical level of abstraction in any way their developers/vendors deem convenient (b) the data represented via logical-level tables.

It is important to note that the physical-level storage can in turn be divided in distinct sub-levels of abstraction.

Therefore, the physical arrangements depend on the particular DBMS under consideration, hence they can vary from platform to platform.

Examples of physical structures utilized by different SQL platforms are detailed

• in the Storage Page Layout content from the PostgreSQL official documentation; and
• in the post entitled Inside the Storage Engine: Anatomy of a record, regarding Microsoft SQL Server, by Paul Randal; and
• in SQLite's Database File Format page.

In case you are interested, you may find of help this answer to a question about some of the different levels of abstraction of a computerized information system.