Once a bootloader has finished it's initialization tasks, it transfers control of the system to the operating program/system. The specific instruction is typically a jump or a branch depending on the specific bootloader or architecture.
Since you specifically mention an operating system, I will reference the Embedded Linux Primer:
Note that the
bootm
command is the death knell for U-Boot. This is an important concept. Unlike the BIOS in a desktop PC, most embedded systems are architected in such a way that when the Linux kernel takes control, the bootloader ceases to exist. The kernel claims any memory and system resources that the bootloader previously used. The only way to pass control back to the bootloader is to reboot the board.
And looking at the AT91 Assembler Code Startup Sequence for C Code Applications Software we can see that it uses bx
which is the Branch and Exchange command of the THUMB Instruction Set:
;--------------------------------------------------------------------------- ;- Branch on C code Main function (with interworking) ;--------------------------------------------------------------------------- IMPORT __main ldr r0, =__main bx r0 END