How did old OSs create or expand a segment in memory without issues?

Question

On an 8086 CPU before the flat memory model had been adopted, when the OS wanted to create a new segment for a process, how did it know what virtual memory ranges were already covered by existing segments; how did it find a suitable base address to use for the new segment descriptor that had suitable room?

Also, imagine one process suddenly requires its segment to expand tremendously and hence its segment limit needs to expand, how is it ensured that the limit doesn't encroach over the base of an adjacent segment, and if there is an adjacent segment, how does it not prevent the growth of the segment, how is this situation handled?

Answer 1

The 8086 does not have a protected mode: every process has full access to everything, which is a nightmare for an OS designer.

The processor had a set of registers specially designed to handle the segments: CS (code), DS (data), SS (stack), and ES (extra spare one). These could then be used in combination with a set of pointer register to address 64K for each segment. However, the code had to handle these under its own responsibility. So compilers adopted typical memory models for generating the code, depending on whether or not the 64K limit could be insufficient for some segments.

The main OS at that time was MS-DOS. No multi-tasking: so no worry about concurrent processes. The programme executable defined the initial memory management:

• the COM executable format was initially used. The OS simply loaded all the code at one address, initialised all the segments to the same 64K bloc, and started the programme at a fixed offset. You then had access to the machine.
• the EXE format later became more popular. It allowed to initialise and load multiple segments at different places. In the header a maximum size allows the OS to reserve initially enough space. The loading process is explained here.

MS-DOS 2 also came with a better memory management. Basically, you could allocate, increase allocation and free blocs of memory from the OS. The blocs were meant to fit in segments. But once again, the program had to manage the loading of segment and pointer register on its own. And nothing protected other parts of the memory from memory corruption.

I don't remember if the realloc at MSDOS level moved the bloc or simply returned an error if there was not enough space to extend the size of the bloc. But the standard C library fortunately provided the nice realloc() that came from the UNIX world, and ensured moving memory to a larger location if needed. Up to you to ensure that no rogue pointer still used the old location.

Answer 2

Until paging virtual memory, operating systems used a single shared address space for both itself and programs.  Using this scheme and with limited available memory, there aren't that many possible approaches to memory management: reserve a fixed size for the initially loaded program, and then additionally, allow the program to allocate non-contiguous memory (to the initial load or other extensions) to satisfy extension requests.

For example, MS-DOS exe format allow the program to specify min & max values for its initial size.  Then it supports memory allocation via int 21h code 0x48. See here, and here.  This works similar to malloc/free/realloc only on global shared program memory rather than within the initial heap.

Also see this retrocomputing SE answer.

DOS would also allow specification, in some cases, whether you wanted a program (or block) placed high or low in memory to allow the programmer to participate somewhat in reducing fragmentation of the shared address space.