Since this question doesn't have any answer that can be marked as Accepted. I am posting the solution.
Looks like adding back the header immediately after the AJAX call would make sense. This way we won't be waiting for success callback and then adding it.
delete $.ajaxSettings.headers["x-custom"]; // Remove header before call
$.ajax({
...
"success": function (data) {
...
}
});
$.ajaxSettings.headers["x-custom"] = 'value'; // Add it back immediately