Under a 64-bit kernel, 16-bit protected-mode user-space is available, but virtual-8086 mode isn't.
Most 16-bit software is written for DOS, and expects to run in real mode. vm86 (virtual-8086) mode is basically hardware virtualization for real mode, allowing the guest to use cli
/ sti
without affecting the real IF bit in EFLAGS for example.
In 16-bit protected mode, cli
would only work if the IO privilege level was 0 (like ring 0), and it would disable interrupts on that actual CPU core, not just inside the 16-bit emulated environment. Thus it's not useful for running 16-bit DOS programs under a modern OS.
So yes, you can run 16-bit user-space code under a 64-bit kernel, but only in 16-bit protected mode which nobody ever uses (AFAIK). I don't think Linux natively supports 16-bit processes, although perhaps you could create a custom 16-bit code segment with the modify_ldt
system call, and jmp far
to it.
It might have been possible to build a software-virtualization system that used 16-bit protected mode to run 16-bit DOS guests, trapping to the kernel/hypervisor when the guest ran instructions like cli
. @Lưu Vĩnh Phúc's answer is one reason that MS didn't try to do that.
Software-virtualization of x86 is non-trivial, though, because some instructions that you need to emulate (like pushf
) don't actually trap. The guest could notice (or break because of) the Interrupts-enabled flag (IF) in the pushf
result not matching a cli
it just ran.
Even the slowest x86-64 CPUs are plenty fast enough to run software like DOSBOX that fully emulates an x86 PC with directly-accessible old hardware, so there wasn't much demand for being able to run 16-bit code natively, or much demand for running it in a way that looked "native", i.e. able to launch other programs under the main OS instead of just inside the emulated environment.
Apparently there was a Linux patch (last updated in 2008) to run vm86 under a 64-bit kernel, maybe by switching the kernel to protected mode temporarily? Or by just running software emulation.
It's possible for a 64-bit kernel to switch back to legacy mode, making it possible to use vm86 mode.
Some kernels in real life actually do run some/most of the kernel in 32-bit mode, perhaps only switching into long mode when 64-bit user-space needs to run, or to read/write RAM that can't be mapped into the limited 32-bit virtual address space.
I think I've read that MacOS did / does this, perhaps to support 32-bit binary kernel drivers.