How to implement programming language in operating system

Question

I recently read some books of operating systems. All of them covered on how to "make" a operating system in detail.

But the problem is, like the operating system, programs are also compiled into machine code, How does the operating system execute them in terms of securities?

Let's say if part of the programs will modify the operating system or other running programs. How does operating system prevent it from this?

Answer 1

The typical solutions are different processor modes. Generally the same instruction set but in different modes there are different protection mechanisms.

For example an MMU is also often used, which does a few things, one it virtualizes memory which one side effect is programs can all think they are in the same address space. All programs can think they are running from address zero for example. Second, the mmu is often used to define cachable and non-cachable regions of memory. And third (but not last) is the mmu is often used to keep a program in its own memory space, if it goes outside that space then a fault occurs and the operating system takes over.

So the operating system itself will run in a super user mode or a mode with the most power and least restrictions. But user code ideally runs in a mode with the most restrictions and it is the operating system that sets up that environment before calling the user code and giving it a time slice. The details are all very processor specific and change from generation to generation and vary from one family/architecture to another.

As far as the programming language that is for the most part unrelated, the compiler doesnt need to and IMO should not know what mode you are in, its job is to convert high level code to lower level code, period. Kernel code will periodically use assembly to assist, for various reasons for example a processor specific instruction that is required that is not directly mapped to a high level programming language. The high level code that surrounds this small amount of asm or calls an asm based function, is from the compilers perspective generic. Now that doesnt mean that some compilers will create compiler specific directives (read: non-portable, sometimes environment or processor specific, ugly) to save the programmer a little bit of assembly language.

Answer 2

Operating Systems on the whole do not care much about languages. It's all executable code to them, and they don't care who made it.

They do care about their own integrity and that of other programs. This is typically fixed by the Virtual Memory mechanism - there's simply no way for a program to form a pointer into memory it doesn't have rights to. Physical memory only has an address if a program is at least allowed to read it.

Answer 3

The exact implementation highly depends on operating system and underlying architecture since some architectures can provide hardware mechanisms for security (for example mark a memory region read-only).