Far return from a 64-bit call gate

Question

In the 64-bit ring 0 environment, I want to switch to ring 3, toolchain: x86-64 gcc 4.6.3, as 2.22. The related code:

#define SHELL64_PHY_BASE    (12 * 1024 * 1024)
#define SHELL64_VIRT_BASE   0x10000000000
#define SHELL64_MAP_SIZE    (4 * 1024 * 1024)

// selector for ring 3 app on 64-bit OS
#define SEL_SHELL64_CODE    (0 | 4 | 3)
#define SEL_SHELL64_DATA    (8 | 4 | 3)

size_t kernel64(size_t par)
{
    map_info_t minfo;

    prints(white, "-------------------------\n");
    prints(OS_INFO "Hello, 64-bit OS!\n");

    // prepare GDT/LDT/TSS/IDT
    update_gdt();
    setup_descriptors();

    // Shell64: 64-bit shell app, map config
    minfo.vaddr = SHELL64_VIRT_BASE;
    minfo.paddr = SHELL64_PHY_BASE;
    minfo.bytes = SHELL64_MAP_SIZE;
    minfo.psize = 0x1000;
    minfo.attr = PATTR_WB | PATTR_U_RW_EX | PATTR_EXIST;
    paging_construct(&minfo);

    dcache_flush();

    sector_read((void *)SHELL64_VIRT_BASE, shell64_sec_start, shell64_sec_count);

    prints(OS_INFO "Will switch to 64-bit shell\n");

    dcache_flush();

    switch_to_shell();

    return 0;
}

void switch_to_shell(void)
{
    __asm volatile
    (
        "sub rsp, 32\n\t"
        "mov qword ptr [rsp + 24], %0\n\t"
        "mov qword ptr [rsp + 16], %1\n\t"
        "mov qword ptr [rsp + 8], %2\n\t"
        "mov qword ptr [rsp], %3\n\t"
        "mov rbp, [rsp + 16]\n\t"
        "mov ds, %4\n\t"
        "mov es, %4\n\t"
        "mov fs, %4\n\t"
        "mov gs, %4\n\t"
        "retf 0"
        : : "i"(SEL_SHELL64_DATA), "r"(SHELL64_VIRT_BASE + SHELL64_MAP_SIZE),
        "i"(SEL_SHELL64_CODE), "r"(SHELL64_VIRT_BASE), "a"(SEL_SHELL64_DATA)
    );
}

GDT info:

+0 null desc
+8 64-bit code desc, non-conforming, DPL 0, for os
+16 data desc, DPL 0, for os
+24 LDT desc
+40 TSS desc

LDT info:

+0 64-bit code desc, non-conforming, DPL 3, for app
+8 data desc, DPL 3, for app
+... the other desc

When the far return instruction(retf) is executed, processor enters an exception. If run debug command 'c', lots of errors appear and crash.

After debug, I found the exception vector is 13, #GP fault. When enter exception, the stack info:

[rsp + 0x00]:00000000 00000100 // error code
[rsp + 0x08]:ffff8000 00001b58 // old RIP
[rsp + 0x10]:00000000 00000008 // old CS
[rsp + 0x18]:00000000 00010082 // RFLAGS
[rsp + 0x20]:ffff8000 000221d0 // old RSP
[rsp + 0x28]:00000000 00000010 // old SS

bochs 2.6.2, Windows version. Host OS is Win8.1

The related crash info: http://pan.baidu.com/s/1kT0qr4r

Answer 1

The problem is as does not generate a REX prefix for a RETF instruction. As such, your code is executing with 32 bit operand size and given that the stack contains SHELL64_VIRT_BASE with value 0x10000000000, it will be interpreted as 32 bit CS:EIP=0x100:0x00000000. In the previous revision of your question you stated that bochs prints the following error:

00012169062e[CPU0] fetch_raw_desciptor: GDT: index (107) 20 > limit (37)

Knowing that the numbers there are in hex, a segment selector of 0x100 would indeed try to access up to byte 0x107 and would have index 0x20.

Solution: In intel syntax you have to manually specify the prefix as rex64 retf or rex.w retf. Using at&t syntax, you can use the lretq mnemonic that will generate the prefix automatically.