Why does malloc overwrite RSP and RSP+8?

Question

You can read about the 64-bit calling convention here. x64 functions are supposed to clean up after themselves however, when I call malloc from .asm, it overwrites the value at RSP and RSP+8. This seems very wrong. Any suggestions?

public TestMalloc

extern malloc : near

.CODE

align 8
TestMalloc proc
    mov rcx, 100h
    000000018000BDB8 48 C7 C1 00 01 00 00   mov         rcx,100h 

    call malloc
    000000018000BDBF E8 CC AC 06 00         call        malloc (180076A90h) 

    ret
    000000018000BDC4 C3                     ret              
    000000018000BDC5 66 66 90               xchg        ax,ax 
TestMalloc endp

END

Answer 1

For the x64 calling convention, even if the parameters are passed in the registers the caller is required to save space for them on the stack:

Note that space is always allocated for the register parameters, even if the parameters themselves are never homed to the stack; a callee is guaranteed that space has been allocated for all its parameters. Home addresses are required for the register arguments so a contiguous area is available in case the called function needs to take the address of the argument list (va_list) or an individual argument.

http://msdn.microsoft.com/en-us/library/ew5tede7.aspx

Answer 2

I'm not sure, truthfully, but have you tried stepping through the assembly in a debugger? If you follow the internal logic you might unearth some clues as to what is going on. I recommend WinDbg.