16-bit assembly incompatibility with 64-bit windows 7

Question 1

To run a “16-bit Application”, which in this case means a DOS application, Windows® needs to set up a task in VM86 mode. The problem is that this works when the CPU is in 32-bit VPAM (Virtual Protected Address Mode), which is what an i386 OS uses, but not when the CPU is in the so-called “Long Mode” that AMD introduced. The amd64 CPU “Long Mode” only supports running 32-bit and 64-bit tasks.

So, a 64-bit OS kernel has no option to run 16-bit tasks directly on the CPU, you always must use some kind of emulation. If you find an OS that can do it, it either has that emulation built in, or is running in 32-bit mode instead of 64-bit mode, or is running in both 32-bit and 64-bit mode and switching happily between them in some kind of evil, twisted hack I've read about somewhere.

From experience, using DOSBOX is your best bet.

Edit: how does Windows® detect that it cannot run your code?

This largely depends on the program type detected. There are batch files (BAT, CMD) which it knows to handle, PIF files (I think they still didn't kill them off), and finally, COM and EXE executables. The COM case is easy: 65280-byte-max. 16-bit MS-DOS® program, out. EXE files, on the other hand, have certain file headers: one for the 16-bit DOS (or Win3.x) part (keyword: MZ), one for the 32-bit/64-bit part (LE, LX, PE (at the least), a.out and COFF are keywords here, some of which are used for OS/2 compatibility or by it only, some by various NT variants).

Question 2

There are actually more than one issue. First of all the processor has different modes of operation for each "bitness". But each type of application requires an execution environment that goes beyond the bitness. It's called an Application Binary Interface (ABI). For example .com files contain code that expects a real mode DOS environment, including certain hardware. Earlier versions of windows needed to fire up something similar to DosBox in order to run them. Every type of executable might not need full blown hardware emulation but a great deal of code to interact with the main operating system. So for every type of executable the operating system will check what kind of runtime environment is needed and refuse to run the executable unless it can provide the right environment.

Question 3

My question is: What exactly makes a code 16, 32 or 64 bit, what triggers the incompatibility problems 16-bit applications apparently have?

Well: the obvious answer would be: depends on that program using 8,16,32 or 64 bit registers. Obvious but not true: a program running in MS DOS can use 8,16, or 32 bit registers if the processor supports them. Alternatively, a Windows program can use (and indeed uses) 8 and 16 bit registers, besides using 32 bit registers and (if the processor and OS supports it) 64 bit registers.

The question, I think, should be: what exactly makes a program to be run under DOS or Windows 32 bits or Windows 64 bits.

First, the structure of the executable file. A 32-bit or 64-bit Windows executable file is not the same as the 16-bit executable file. Windows uses the so called "Portable Executable" (short for PE). MS DOS uses both .COM files (inherited from CP/M) and .EXE files (also known as MZ executables).

Second: one the program is stored in memory and begin executing, there some environment that the program expects to be there, for example, an army of function calls accessed by special instructions, like INT, which are in charge of performing different task, from outputting a character to the console, to open and read files, and many more.

Third: MS DOS applications expects some structures in memory to be at a specific memory address; for example the screen memory. Also it expects some I/O devices to exist at a specific I/O address. Many so called "bad behavioured" applications don't use BIOS or DOS to print on screen, but write directly to screen memory, for faster output. I remember a setting in Turbo Pascal, Turbo Basic, or maybe Turbo C compiler, to let the programmer choose console output functions to go through BIOS, or to go directly to hardware.

Fourth: the program expects the processor to behave in a particular way. 16-bit DOS programs are designed to use the so called Real Mode of x86 processors, and this mode imposes its particular vision of memory addresses, to name the most distinct feature of this mode. With the arriving of 32-bit OS's that puts the processor in Protected Mode, real mode programs cannot work because the way memory addresses work in protected mode is very different from the way it worked for real mode.

An OS willing to run MZ executables need some sort of loader for these files (to comply with the first point) and an environment that has to mimic the one the application should work into (to comply with the rest of points). In the Linux world these components are part of what they call personalities (a way for Linux to run code from other OS's such as FreeBSD, providing that code is intended for the same processor on both platforms). In the Windows world this is known as subsystems. Windows XP and Windows 7, for example, have the Win32 subsystem and the DOS subsystem. Ancient versions of Windows (Windows NT 3.5 or NT 4.0 perhaps) had an OS/2 personality too, capable of running OS/2 16-bit text mode applications.

So, your Windows OS need the appropiate subsystem to run code designed for another OS like MS DOS. But the processor must help in any possible way. This is because the x86 platform introduced the 8086 virtual mode in the 80386: to be able to create a task in which the processor behaves much like a real mode 8086.

So with assistance from the CPU itself, the appropiate loader, an INT handler capable of translating DOS request into Windows request, an intelligent page handler that is able to catch accesses to specific sections of memory and translate them into requests for drawing characters on screen, and a I/O fault handler capable of impersonate the devices that the application used to work with, a program written for DOS can run under Windows, or Linux (DOSemu)

So, what happens with 64 bits? For a code to be able to run with access to 64-bit registers, another kind of protected mode is employed, called long mode. I don't know much about this long mode, but from what I have read, this mode doesn't allow the processor to create a task using the 8086 virtual mode. There is another mode called compatibility mode or something like that, that allows the processor to create 64 and 32-bit tasks, but no 8086 virtual mode tasks.

Without CPU assistance, a DOS subsystem is much difficult to achieve: you could mimic the memory map, trap any access to special memory regions or I/O, trap INT instructions, but besides, you should trap any attempt to update the segment registers in order to try to mimic the real mode addressing as well, and I'm not actually sure segments are in use in long mode. It's doable, as applications like DOSBox prove it, but Microsoft doesn't need DOS anymore (it hasn't need it for a long time, but customers didn't share their point until, perhaps, the introduction of Windows 2000/XP). Windows 7 32-bits can use the same DOS subsystem as previous Windows versions used (I even read some time ago about a hack to bring the OS/2 subsystem back into Windows 2000 or Windows XP), but for Windows 7 64-bits, a complete new DOS subsystem had to be designed (or bought).

More info here: http://www.codeproject.com/Articles/45788/The-Real-Protected-Long-mode-assembly-tutorial-for#Main

Question 4

The general approach is to use an emulator, such as DosBox. There is not a way to change something about a binary to make it run using a different instruction set. That would have to be done starting from the source code.

Question 5

Install DOSBox and have fun. It's free at http://www.dosbox.com/

EDIT:

The bitness of the code means assumptions about register/operand size. The current "bitness" of the CPU, on the other hand, is 16 in real mode, and in protected mode, it's a flag in the in-memory structure that describes the currently executing piece of code (segment descriptor and/or page entry, to be precise). The OS sets up said memory metadata and maintains the bitness. Depending on the latter, the same command bits might mean

add ax, bx

or

add eax, ebx

or

add rax, rbx

The bitness of the CPU and the assumptions of the code better match. If they don't, it won't take long before the application crashes or misbehaves. As soon as the code tries to load, say, a register from memory and the CPU-level register size is not what the code thinks it is, the CPU would load garbage (or fail to load a portion of the register). Just imagine what would happen to the following in 32-bit mode:

MyData dw 0x17 ;2 bytes of data
SomeOtherData dw 0x200 ;2 more
...
mov ax, [MyData] ;We think we mean AX, but the CPU thinks it's EAX

The EAX would load the value of 0x2000017. Definitely not what the program expects. Now imagine the program then compares the value to 0x17 and does a conditional jump based on that.

It's possible to translate 16-bit real mode assembly to 32-bit protected mode by hand, but it would not just be translating the register names. From a DOS program you'd have to go to a Windows program, and those are a different bunch. Not a straightforward translation - more like porting.

Question 6

It's purely a commercial decision - there is simply no technical reason, as is proved by the fact that your machine itself could run a 64-bit or 32-bit version of windows7 - your choice, and you could then run Microsoft's XP emulator which WILL run .COM files.

Microsoft clearly has the technology available - it's described HERE but don't allow it to run on Home* editions.