What enforces memory protection in an OS?

Question

In general, I know that a process can't write to a memory (in its addresses space) that has a protection that doesn't allow writing. But what checks whether the process can do this? Does any assembly instruction goes through the operating system? how does it work?

Answer 1

In most modern CPUs (Intel x86, most ARM flavors) it's the CPU itself that does the checking. The CPU stores, in one of the registers, an address of a data structure that specifies the layout of the memory ("page table") - specifically, which addresses are readable, which are writable, which are executable. Every memory accessing operation in the CPU is checked against the page table.

When a program tries to do something to a memory location that the respective page table entry does not allow, the CPU generates an exception (interrupt), and the OS gets control. Further actions depend on the OS. One common scenario involves the OS displaying an error message and terminating the faulty program. Not necessarily, though. For example, page swapping (writing memory out to a page file on a disk and reading back when needed) is implemented via the same mechanism.

The page table is maintained by the OS and is not (typically) visible to userland code. The relevant portions in the OS are hardware dependent.

Answer 2

The page tables, they have all the information for the currently executing process' memory space. When you try to access memory that is either read only by writing to it, or accessing memory that doesn't belong to you the processor fails to find a mapping (or sees you cant write to it) and issues a page fault to the OS. The OS then decides whether it is a copy on write page, if the page you accessed belongs to you at all but hasn't been mapped yet, or if you just accessed bad space and handles it accordingly (usually kills the process with a segfault if you access bad space).

Answer 3

When this is supported in hardware, typically there is a bit in the information describing a block of allocated memory that indicates whether the block of memory may be executed.

On Intel processors this is called the NX (Never Execute) bit, while AMD calls that the XD (Execution Disabled) bit.

The NX bit specifically refers to bit number 63 (i.e. the most significant bit) of a 64-bit entry in the page table. If this bit is set to 0, then code can be executed from that page; if set to 1, code cannot be executed from that page, and anything residing there is assumed to be data.

http://en.wikipedia.org/wiki/NX_bit

The bit is set by the operating system after it loads executable code into memory. It may only be set by privileged code (e.g. the OS, or drivers).

See also:

http://en.wikipedia.org/wiki/Executable_space_protection