I have been informed of a workaround to this issue whereby you call:
challengeHandler.activeRequest.onUnresponsiveHost()
Before actually calling:
challengeHandler.submitFailure()
This will trigger the failure and reject the deferred as intended.
This is a workaround though so it might not work in future versions. I am going to raise a request for enhancement (link pending) and I would appreciate it if you could vote it up as well to ensure this functionality is available in the future.