C alloca function - what happens when too much memory is tried to be allocated

Question

In C the alloca() function allocates memory on the stackframe of the caller of alloca().

What happens when you try to allocate a huge number of bytes that it cannot possibly allocate?

• Does it allocate as many bytes as it can until the stack meets the heap segment?

• Or does it allocate nothing at all?

  alloca( 100000000000000000000 );

The manual mentions:

The alloca() function returns a pointer to the beginning of the allocated space. If the allocation causes stack overflow, program behavior is undefined.

I understand the behaviour is undefined. But there must be more to say than that:

• What does it return, a pointer to the first byte after the top of the stack before main was called?
• After alloca() returned is the stack pointer different then what it was before alloca() was called?

Does anyone have some more info about this?

Answer 1

The GNU libc implementation of alloca() reads:

# define alloca(size)   __builtin_alloca (size)

It uses a compiler built-in, thus it depends entirely on how it's been implemented by the compiler. More specifically, it depends on how the stack is handled, which happens to be a a machine and ABI-dependent data structure.

---

Let's examine a concrete case. On my machine, this is the assembly for alloca(100000000000L):

0e9b:  movabsq $-100000000016, %rax ; * Loads (size + 16) into rax.
0ea5:  addq   %rax, %rsp            ; * Adds it to the top of the stack.
0ea8:  movq   %rsp, -48(%rbp)       ; * Temporarily stores it.
0eac:  movq   -48(%rbp), %rax       ; * These five instructions round the
0eb0:  addq   $15, %rax             ;   value stored to the next multiple
0eb4:  shrq   $4, %rax              ;   of 0x10 by doing:
0eb8:  shlq   $4, %rax              ;   rax = ((rax+15) >> 4) << 4
0ebc:  movq   %rax, -48(%rbp)       ;   and storing it again in the stack.
0ec0:  movq   -48(%rbp), %rax       ; * Reads the rounded value and copies
0ec4:  movq   %rax, -24(%rbp)       ;   it on the previous stack position.

Compiled using gcc-4.2 -g test.c -o test from the following program:

With something to refer to, your questions can now be answered:

Does it allocate as many bytes as it can until the stack meets the heap segment?

It just blindly grows the stack by the requested number of bytes. No bounds checking is performed at all, so both the stack pointer and the returned value will probably now be in a illegal location. Attempting to read/write from the returned value (or push onto the stack) will result on a SIGSEGV.

What does it return, a pointer to the first byte after the top of the stack before main was called?

It returns a pointer to the first byte of allocated memory.

After alloca() returned is the stack pointer different then what it was before alloca() was called?

Yes, see explanation above. Also, when the function that called alloca returns, the stack will be restored to previous frame and it will be usable again.

Answer 2

What happens depends on your compiler and the hardening options in use; typically, you have no indication that it failed and you either clobber random unrelated memory, or crash, shortly after calling alloca. With some hardening options you may be able to make the crash reliable, but you're never able to detect and recover from the failure. alloca simply should not be used. It was a bad mistake that looked too good to be true, because it is.

Answer 3

Strictly speaking, nobody knows, because "undefined behavior" isn't itself defined. (alloca isn't defined by the C or POSIX standards, for example).

For illustration only, C's definition of "undefined behavior" is (ISO 9899:1999, section 3.4.3):

"behavior, upon use of a nonportable or erroneous program construct or of erroneous data, for which this International Standard imposes no requirements

"NOTE Possible undefined behavior ranges from ignoring the situation completely with unpredictable results, to behaving during translation or program execution in a documented manner characteristic of the environment (with or without the issuance of a diagnostic message), to terminating a translation or execution (with the issuance of a diagnostic message)."

So: absolutely anything can happen. Your hard disk might get reformatted. The sky might fall in. (OK, probably not, but it would be perfectly acceptable given your input.) You can't make any assumption or statement whatsoever.

If your program makes any such assumptions about (or reliance on) program behaviour after an alloca-induced stack overflow then your program is broken. It's best not to surmise what a particular compiler might do in that situation. Your program's broken, end of story.

Answer 4

On windows you can recover from it. Tested using gcc:

/*
 * Show how get memory from stack without crash
 * Currently, compiles ok with mingw, and the latest version of tiny c (from git)
 * Last Version: 29/june/2014
 * Programmed by Carlos Montiers
 */

#include <windows.h>
#include <stdio.h>
#include <stdlib.h>
#include <limits.h>
#include <malloc.h>
#include <setjmp.h>

int _resetstkoflw(void);

static jmp_buf alloca_jmp_buf;

LONG WINAPI AllocaExceptionFilter(EXCEPTION_POINTERS * ExceptionInfo)
{

    switch (ExceptionInfo->ExceptionRecord->ExceptionCode) {
    case STATUS_STACK_OVERFLOW:

    // reset the stack
    if (0 == _resetstkoflw()) {
        printf("Could not reset the stack!\n");
        _exit(1);
    }

    longjmp(alloca_jmp_buf, 1);

    break;
    }

    return EXCEPTION_EXECUTE_HANDLER;
}

int main()
{
    void *m;
    int alloca_jmp_res;
    LPTOP_LEVEL_EXCEPTION_FILTER prev;

    //replace the exception filter function saving the previous
    prev = SetUnhandledExceptionFilter(AllocaExceptionFilter);

    alloca_jmp_res = setjmp(alloca_jmp_buf);
    if ((0 == alloca_jmp_res)) {
    m = alloca(INT_MAX);

    } else if ((1 == alloca_jmp_res)) {
    m = NULL;

    }
    //restore exception filter function
    SetUnhandledExceptionFilter(prev);

    if (!m) {
    printf("alloca Failed\n");
    }

    printf("Bye\n");
    return 1;

}