Structured Exception Handler (SEH) does not catch heap corruption

Question

I'm writing small utility (VC 2010, no clr) that does one simple task (rasterizing) using 3rd party library. Later utility will be used by bigger application. Sometimes the utility crashes because of some heap corruption in 3rd party library. That is OK, but Windows (Vista/2008) shows well known dialog "Program has stopped working ... Close/Debug program." which is not appropriate in my case (Server side). Utility should crash/terminated silently w/o any visible effects.

To do that I installed SEH for unhandled exception (SetUnhandledExceptionFilter). The handler is perfectly invoked for exceptions like AV ( *(PDWORD)0 = 0 ), but for some reason it is not invoked in the case of heap corruption. Corruption happens in dllmain of one of 3rd party library dlls while it is being unloaded.

Couple of questions. Can anybody explain why the handler is not invoked? Is there any another way to prevent that dialog?

Answer 1

Apparently it's intentional that heap corruptions cannot be caught by user-defined exception handlers, even though they're emitted as exceptions with their own exception code (0xC0000374 "STATUS_HEAP_CORRUPTION"). Here's a Visual C++ bug report which was basically closed as "won't fix":

https://connect.microsoft.com/VisualStudio/feedback/details/664497/cant-catch-0xc0000374-exception-status-heap-corruption

As you have discovered, this is not a bug in the compiler or the OS. The heap corruption that your function causes is treated as a critical error, and as part of handling that error the OS terminates the process. This is what causes your exception handlers to not be invoked.

I'd guess that Windows Error Reporting or other ways of creating a crash dump could still catch it.

As for preventing the dialog, in the registry you can either disable WER completely or just disable the dialog so that the process won't block:

https://msdn.microsoft.com/de-de/library/windows/desktop/aa366711(v=vs.85).aspx (see "DontShowUI")

Answer 2

but for some reason it is not invoked in the case of heap corruption. Corruption happens in dllmain of one of 3rd party library dlls while it is being unloaded.

Heap corruption is undefined behavior. It may throw exceptions, it may do otherwise. If a buggy third party library is messing up your heap, then the question is "why are you allowing them to mess with your heap in the first place?"

The "program has stopped working" dialog is shown whenever a process terminates abnormally. Not all abnormal process terminations result from exceptions. Many errors (such as stack overflow, misaligned stack, etc.) cause instant termination of the process, which may show that message but will not give you a chance to handle the error.

(Also, see Hans' awesome comment above)