As it's windows forms app Application.ThreadException is used for unhanded exception: "This event allows your Windows Forms application to handle otherwise unhandled exceptions that occur in Windows Forms threads. Attach your event handlers to the ThreadException event to deal with these exception" (MSDN)
I think you must expect such behavior as at UIThreadExceptionHandler you have Application.Exit(new CancelEventArgs(true)). Exit methods description: "Informs all message pumps that they must terminate, and then closes all application windows after the messages have been processed." (MSDN)
AppDomain.CurrentDomain.UnhandledException event is used to handle non-UI thread exceptions.
EDIT 1:
AppDomain.CurrentDomain.UnhandledException is specifically designed to log an exception before the system will inform a user and terminates the process. You can not prevent terminating the process (unless you are using Net 1.1).
Application.ThreadException + UnhandledExceptionMode.CatchException allows you to keep your UI thread alive. But, it's really not a great idea. It's much better to enclose error prone code in try-catch blocks instead.
Thus if you want to catch exceptions coming from thread and keep your application alive, you have to do it inside using try-catch block.
You haven't issue with UnhandledExceptionHandler, because it's not fired at all, I suppose.