how does linux kernel prevents the BIOS system calls?

Question

BIOS calls are not available in linux OS. I wonder how does a kernel prevents execution of such instructions that contains call to the BIOS subroutines?

Answer 1

The INT n instruction generates a call to the interrupt or exception handler specified with the destination operand. The destination operand specifies an interrupt vector number from 0 to 255, encoded as an 8-bit unsigned intermediate value. Each interrupt vector number provides an index to a gate descriptor in the IDT.

The selected interrupt descriptor in turn contains a pointer to an interrupt or exception handler procedure. In protected mode (linux works in protected mode only), the IDT contains an array of 8-byte descriptors, each of which is an interrupt gate, trap gate, or task gate.

This IDT is set by the OS. Linux sets it up so that descriptors point to its own handlers, not the BIOS handlers at all.

Answer 2

The BIOS is mostly available in 16 bits mode, not in the 32 or 64 bits mode of x86 on which Linux runs.

A Linux process is running in user mode, using virtual memory, it has its own virtual address space.

Some machine instructions (notably INT which is used to go to BIOS) are privileged so cannot be run in user mode. If you try running them in user mode, the processor makes a machine exception, and the kernel handles it by sending some signal. (some INT is also used for syscalls, but the SYSENTER instruction is preferred).

An application uses syscalls to interact with the kernel (perhaps thru the VDSO).

Read the assembly howto. Ask also on kernelnewbies.org

The Linux kernel is open source, you are allowed to download it and study its source code from kernel.org

PS. After 2020 be aware of UEFI. It is a complex thing.