Assuming you're using standard unobtrusive/jQuery validate; Disable client-side validation by putting a class of "cancel" on the button:
<button type="submit" class="cancel">Delete</button>
This will prevent client-side validation from firing at all in the event of this button being clicked.
For server side, just don't check if the model's valid or not.