How to increase c++ thread stack size on solaris using pthreads?

Question

What is the easiest way to increase the size of the default stacksize for pthreads?

You can use pthread_attr_setstacksize to set the stacksize when creating new threads. The stack size must not be smaller than PTHREAD_STACK_MIN.

Is there any way to tune the heapsize as well, process level and individual thread level?

Using the Solaris compiler you can try to change the pagesize using the -xpagesize option, but you cannot adjust the size of the heap (it will be as large as the memory available to the machine). There is only one heap shared by all threads, so you cannot adjust it per-thread.

If new operator fails because of underlying memory leakage, how do I set new handler to take care of bad allocations?

The new handler is a specialized feature and there is no general answer, how to use a new handler is very dependent on the details of your program. It can't be used to fix memory leaks, once the memory is leaked it's too late so you need to prevent leaks from happening in the first place. (And if you don't know how to write a new handler then you probably don't need to use one.)