Sounds like a stack overflow, alloca will allocate from the stack and so if you attempt to allocate too much that will result in a stack overflow while malloc will allocate from the heap which is much larger. There is no way to know when you overflow the stack with alloca.
Also, note that since alloca
allocates on the stack returning that memory from the function won't work since that memory will no longer be available once you exit the function. So if you need to use the memory outside the function you will need to use malloc
.