It's possible to define your own stack, although it's hard to do in a completely portable fashion. In principle, you can do it with posix threads (see pthread_set_stackaddr
) so its not even particularly exotic.
Then you can use mprotect
to create a red zone ("guard area") at the end of your stack. Stack overflow will then result in a SIGSEGV
, which you can handle; the sigaction handler will be provided with the address of the protection fault, so you can check in the handler to see if its part of one of your red zones. (Of course, you have to run the signal handler with an alternate stack. See the SA_ONSTACK
flag and sigaltstack
.
Recovering from stack overflow (without compiler assistance) is trickier.
By the way, there is no relationship between the size of a function (as a relocatable object consisting of a series of machine instructions) and the size of a function's stack frame. So nm
really isn't going to do you any good.