Application.Exit
will cause your form to be closed (recursively) so you see the message box again. In the event the user presses Yes in the message box, you should just do nothing in your event handler and allow the application exit to continue.
By not setting e.Cancel = True
you would indicate that you want the form shutdown to continue.