How to change the stack size using ulimit or per process on Mac OS X for a C or Ruby program?

Question 1

Apparently there is a hard limit on the stack size for mac os x, taken from http://lists.apple.com/archives/scitech/2004/Oct/msg00124.html granted this is quite old, and Im not sure if its still true anymore, but to set it simply call ulimit -s hard, its 65532. or about 65 megs.

I did some tests on snow leopard, 10.6.8, and it does seem to be true.

$ ulimit -a
...
stack size              (kbytes, -s) 8192
...
$ ulimit -s 65533
-bash: ulimit: stack size: cannot modify limit: Operation not permitted
$ ulimit -s 65532
$

I also found this http://linuxtoosx.blogspot.com/2010/10/stack-overflow-increasing-stack-limit.html though I haven't test it, so can't really say much about it.

When applications consume gigs of memory thats usually taken from the heap, the stack is usually reserve for local automatic variables that exist for a relatively small amount of time equivalent to the lifespan of the function call, the heap is where most of the persistent data lives.

here is a quick tutorial:

#include <stdlib.h>

#define NUMBER_OF_BYTES 10000000 // about 10 megs
void test()
{
   char stack_data[NUMBER_OF_BYTES];          // allocating on the stack.
   char *heap_data = malloc(NUMBER_OF_BYTES); // pointer (heap_data) lives on the stack, the actual data lives on the heap.
}

int main()
{   
    test(); 
    // at this point stack_data[NUMBER_OF_BYTES] and *heap_data have being removed, but malloc(NUMBER_OF_BYTES) persists.
    // depending on the calling convention either main or test are responssible for resetting the stack.
    // on most compilers including gcc, the caller (main) is responssible.

    return 0;
}

$ ulimit -a
...
stack size              (kbytes, -s) 8192
...
$ gcc m.c
$ ./a.out
Segmentation fault
$ ulimit -s hard
$ ./a.out
$

ulimit is only temporary you would have to update it every time, or update your corresponding bash script to set it automatically.

Once ulimit is set it can only be lowered never raised.

Question 2

To my mind the accepted answer is not totally right and leads to miss comprehension, more specifically the last statement is not true.

Once ulimit is set it can only be lowered never raised.

There are indeed soft (displayable with ulimit -s or ulimit -Ss) and hard (displayable with ulimit -Hs) limits. But while setting the limit through ulimit -s will affect soft and hard values.

Once hard limit is set it can only be lowered never raise, but soft limit can be lowered or raised provided that the value stays lower than the hard limit.

This will work:

# base values
$ ulimit -s
100
$ ulimit -Hs
100
$ ulimit -Ss
100
# lower soft limit only
$ ulimit -Ss 50
$ ulimit -s
50
$ ulimit -Hs
100
$ ulimit -Ss
50
# raise soft limit only
$ ulimit -Ss 100
$ ulimit -s
100
$ ulimit -Hs
100
$ ulimit -Ss
100
# lower soft and hard limit
$ ulimit -s 50
$ ulimit -s
50
$ ulimit -Hs
50
$ ulimit -Ss
50
# then impossible to raise soft limit due to hard limit
$ ulimit -s 100
-bash: ulimit: stack size: cannot modify limit: Operation not permitted
$ ulimit -Ss 100
-bash: ulimit: stack size: cannot modify limit: Invalid argument

Question 3

The system default stack size varies from different version of kernel to kernel. My 10.7 is 16384, such that ulimit -s 16384 is accepted by my Mac. You can try sysctl kern.stack_size and it shows the read-only stack size. mine is 16384.
You can see this technical article, http://developer.apple.com/library/mac/#qa/qa1419/_index.html, to see how to change the default stack size for C program. For Ruby, because it's a scripting language, you have to enlarge its stack size during linking Ruby interpreter. Excepting for having very deep function calls or recursion, or having very large array and objects being allocated in the stack, your program should not have huge stack space. Instead, using heap or dynamically allocation can use up to 2GB of RAM as you wish.

Question 4

I found that using /bin/zsh instead of /bin/sh made this error go away.

For me, the error was occurring in a shell script that called ulimit -s unlimited. When the script was interpreted by /bin/sh (i.e., had #!/bin/sh as the first line of the script file), it barfed with this error. In contrast, when changing it to use zsh, everything seemed to work fine. zsh was smart enough to interpret unlimited as "give me the largest limit the operating system will let me have", and everything worked as you'd want it to.

Question 5

All the limits that the builtin ulimit controls are actually implemented in the OS kernel and as such you should see the C interface documentation for the whole system. Here's Apple documentation for setrlimit(): https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man2/setrlimit.2.html

(Note that the path of that document seems to say iPhoneOS but the content still speaks about "Mac OS X". If you have suitable documentation installed locally, running man setrlimit in your terminal should emit the up-to-date documentation.)

Newly created processes inherit the limits from the fork() parent or the previous process that executes exec().