I had the same problem (but there is a work around for it). You could put a flag into the http context and access it later in the web api and handle it there. You cannot throw an exception from the custom serializer and capture it in the web api. It just ignores the exception and forces to the execution to the end of the ReadJson function if you are trying to over ride this.
you can use this for http context HttpContext.Current.Items["Error Info"] = "Some Error Info”
public override object ReadJson(JsonReader reader, Type objectType, object existingValue, Newtonsoft.Json.JsonSerializer serializer){
...
throw new Exception();
...
///comes here to the end directly
}