To be clear, you want to restart the app if it crashes, but you don't want to restart it if you close it manually. Did I get this right?
I'm not an expert on this, but my guess is that specifying a custom exception handler is delaying the system from wiping out your app until it can call onDestroy(). This means that you go through the same lifecycle regardless of how your Activity ends.
Perhaps there's a way to get around this. You can detect that your app is finishing by calling onFinishing() in onPause(). If you disable the customExceptionHandler at that point, it won't try to restart the app.