Structure of stack of a function call

Question

I've been reading Aleph One's paper on Smashing the Stack for Fun and Profit. I wrote down example1.c from his paper, modified it a bit to see what the stack looks like on my system.

I'm running Ubuntu (64-bit) on a VM on an Intel i5 M 480.

The paper says that a stack will have the following structure. It also says that the word size is 4 bytes. I read up on word sizes and determined that on a 64-bit OS that is not "long-enabled" the word size is still 32 bits or 4 bytes.

However, when I run my custom code:

void function(int a, int b, int c) {
    char buffer1[5];
    char buffer2[10];
    memset(buffer1, "\xaa", sizeof(buffer1));
}

void main() {
    function(1, 2, 3);
}

I do not get the same stack structure as the paper. Yes, I'm aware that the paper was published in 1998 but I haven't found any article on the internet stating that the stack structure has been modified greatly. Here's what my stack looks like (I'm also uploading GDB screenshots for verification, in case I've misinterpreted the stack):

    Lower memory                                                Higher memory
    -------------------------------------------------------------------------   
    | int c   | int b   | int a   | buffer1  | buffer2  | RBP     | RET     |
    | 4 bytes | 4 bytes | 4 bytes | 16 bytes | 16 bytes | 8 bytes | 8 bytes |
    -------------------------------------------------------------------------

Now for my questions:

1. Why has the stack structure changed?
2. What is with the extra space given to buffer1 and buffer2? According to the paper they should have only 8 bytes and 12 bytes allotted. However, buffer2 gets an extra 6 bytes and only then does buffer1 begin and even buffer1 is allotted 16 bytes. Am I missing something here? I read about slack space being given as a protective mechanism, is this it?

[EDIT]

The x64 ABI says that the stack is always eightbyte aligned (page 17, footnote number 9).

So, if the parameters and the 2 character buffers are pushed onto the stack then the total space occupied by the stack is 48 bytes (or 0x30 bytes). Since, it is 8 bytes for 3 integers amounting to a total and 24 bytes and 8 bytes for buffer1 and 16 bytes for buffer2, we arrive at our grand total of 48 bytes which is corroborated by the GDB screenshot.

However, the GDB stack also shows that the integers (a, b and c) have only been allotted 4 bytes each. The buffers also occupy the size declared in the program (obviously). Is this why I see the slack?

Slack = 48 - (5 + 10 + 4 + 4 + 4) = 21 bytes

Am I right?

Answer 1

The difference is mostly explained by the difference of ABI :

• the paper explains what happens in x86 (32 bit);

• you are using x86_64.

In x86n the x86 ABI is used

• The parameters are passed on the stack by the caller. This means that the 3 values on the top will be a, b and c.

• The call instruction pushes the IP (the ret field).

• Usually, the caller frame pointer is pushed by the callee (the sfp field).

• All those fields are fixed on this architecture by the ABI.

• The lower part of the stack (local variables, saved registers, etc.) is not fixed by the ABI and the compiler can decide how it will use it.

Lower memory                                                Higher memory
-------------------------------------------------------------------------   
| buffer2 | buffer2 | sfp     | ret      | a        | b       | c       |
-------------------------------------------------------------------------
<----------(3)------><--(2)--><--------------------(1)------------------>

(1): Pushed by the caller for the callee, fixed by the ABI.
(2): Pushed by the callee, fixed by the ABI.
     Technically, this is optional as for x86 but it is generally used.
(3): Pushed by the callee, in any order.

In x86_64, the AMD-64/x86_64 ABI is used:

• Parameters are usually passed by registers: the a, b, c variables that you find on the stack are copies of the parameters made by the callee (because you probably did not enable optimisations). This is why they are lower in the stack than the return address. This means that the compiler if free to place them in any order (and does not need to have them on the stack at all).

• Moreover, the code is usually not compiled with frame pointers: the push rbp; mov rbp, rsp is usually omitted.

Lower memory                                                Higher memory
-------------------------------------------------------------------------   
| int c   | int b   | int a   | buffer1  | buffer2  | RBP     | RET     |
| 4 bytes | 4 bytes | 4 bytes | 16 bytes | 16 bytes | 8 bytes | 8 bytes |
-------------------------------------------------------------------------
<-------------(2)--------------------------------------------> <---(1)-->

(1): Pushed by the caller for the callee, fixed by the ABI.
(2): Pushed by the callee, in any order.

In summary:

• On x86, the parameters are pushed on the stack by the caller: their locations on the stack is fixed by the ABI. They are higher than the return address because they are pushed by the caller.

• On x86_64, the parameters are passed in registers (unless you have too many of them). The callee is free to push them on the stack if it needs to in any order. As they are pushed by the callee they are lower than the return address on the stack (and can be intermixed with local variables, saved registers).

• In both ABIs, using %rbp as frame base is optional but it is usually used for x86 and often not used in x86_64.

Note: the Windows x86_64 ABI is different.

Some references:

• stack layout in x86_64;

• stack layout in x86.